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ABSTRACT 
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"Preface" (Harish Rathore, Rudolf Schuemer) defines evaluation in DE and 
provides an overview of the evaluation philosophies, agendas, and 
methodologies discussed in the five papers. "Evaluation Practices at the 
Indira Gandhi National Open University" (Krishnapillai Murugan) considers 
evaluation for program design/development, evaluation of student entry and 
performance, and program evaluation. "Evaluation at National Open University 
of Venezuela" (Leopoldo Machado, Arlette Machado) focuses on evaluation of DE 
instruction and administration. Evaluation philosophy, methods, and 
procedures are described in "The Evaluation of Distance Education at the 
University of Victoria" (Geoff Potter) . Program, student, and systems 
evaluation and evaluation criteria are the subject of "Quality Assessment and 
Evaluation: Basic Philosophies, Concepts and Practices at NKI, Norway" 
(Torstein Rekkedal) . "Programme Evaluation at the British Open University" 
(Alan Woodley) covers methods of system evaluation and formative and 
summative course evaluation (critical commenting, developmental testing, 
feedback from tutors and students, cross-sectional studies, and developmental 
studies) and evaluation agendas (market, liberal, and radical evaluation) . 
Most papers contain substantial bibliographies. (MN) 
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Preface 



Some years ago the ZIFF published a report on Evaluation Concepts and Practice in 
Selected Distance Education Institutions. The objective of this earlier report was 
characterized as follows: "This report is neither a textbook nor a treatise on the general 
principles of evaluation in distance education, nor is it a report on the results of single 
concrete inquiries into evaluation on distance education or on components of distance 
education. It is rather a report situated on a medium level of abstraction. Authors who have 
been working in the field of evaluation have been given the opportunity to present the 
concepts and practices of evaluation at their own institution. “ (Schuemer 1991, pp. 1-2). 

In the last few years, distance education has undergone many changes - especially due to 
the use of the "new media". It does therefore seem appropriate to publish an updated 
version of the earlier report. Thus we asked colleagues from different institutions to describe 
the evaluation agenda and methods at their institution. 



On the following pages, we will comment generally on the papers without going into them in 
detail. In the course of this preface, we will briefly examine the definition of evaluation, the 
evaluation philosophy and concepts, and the evaluation agenda and methods. 



Definition of Evaluation in Distance Education 

Just as in other areas of evaluation, there is no universally valid definition of evaluation in the 
field of distance education 1 . 



For the purpose of this report, however, Mary Thorpe's definition of evaluation in the area of 
open and distance learning may be taken as a basis: "Evaluation is the collection, analysis 
and interpretation of information about any aspect of a programme of education and training, 
as part of a recognised process of judging its effectiveness, its efficiency and any other 
outcomes it may have." (Thorpe 1988, p. 5). 



This definition is comprehensive enough to encompass different approaches, concepts, 
tasks, and methods of evaluation. Similarly extensive concepts and definitions of evaluation 
will be found explicitly in some of the contributions. Even in those articles in which there is no 
such explicit definition, a similarly comprehensive concept is apparently implied. 



Evaluation “ Philosophy “ 

Obviously, the authors of the essays do not consider evaluation as an end in itself. As a form 
of quality control with regard to definite application and utilization contexts, evaluation should 
rather provide information which can be used to maintain or to improve the quality of 
products and processes. Thus the meaning of evaluation which is expressed in the articles is 
mainly characterized by considerations of utility as typical for applied research. (On the other 
hand, this does not mean that methodological considerations and standards are not duly 
taken into account, which for instance becomes very evident in the article of Rekkedal.) 



1 For the lack of a universally valid concept of evaluation in distance education cf. e.g. Schuemer (1991, p. 9), Holmberg (1989, 
p. 170), or Holmberg (1995, p. 1 83); see also Woodley's paper in this report; for the lack of a uniform concept for the area of 
programme evaluation cf. among others Wottawa & Thierau (1990). 
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Given the predominantly pragmatic orientation and the variety of purposes which are to be 
fulfilled by evaluation in each of the institutions (see below), none of the papers will restrict 
itself to only one distinct and clear-cut evaluation model. Rather, several articles refer to 
different philosophies, approaches and strategies of evaluation. In this respect, e.g. 
Woodley’s paper refers to three different types or traditions of evaluation (market evaluation, 
liberal evaluation, radical evaluation), which exist simultaneously at the British OU and 
occasionally come to fruition in one and the same study, while Rekkedal gives examples of 
applied evaluation which are based on very different paradigms such as the paradigm of 
"classic" experimental research (in the tradition of Campbell) or on Stake's paradigm or 
concept of 'responsive evaluation’. 

With regard to the subdivision of evaluations into formative and summative ones (according 
to Scriven 1967, 1980), most of the evaluation applications being presented in the single 
papers are formative ones: mostly, evaluation is in the main directed at developing the 
products and processes which are to be evaluated rather than at finally assessing their 
quality and their effects. 

The evaluation philosophy and concepts employed by a given institution are apparently 
influenced by the history and the tradition of this institution. Institutions which had been 
conceived as distance education institutions from the start may have developed a different 
understanding of evaluation to those institutions which initially had been conventional (on- 
campus) institutions which added distance education courses in the progress of their 
development. In this respect, Potter's article, which deals with evaluation at The Victoria 
University, a dual-mode institution, refers to the fact that the concepts of evaluation at this 
institution have developed out of the 'university's traditional, classroom-based instructional 
history’. 

Whatever the details of evaluation philospophy in the respective institutions might be; there 
is one thing which becomes evident in all the articles: evaluation is (more or less explicitly) 
regarded as an area of activity which finally serves to harmonize the programme of an 
institution with the learners’ needs. 



Evaluation "Agenda" 

The articles comprise an impressive variety of evaluation tasks which are completed by 
means of rather different methods and procedures (on the variety of evaluation tasks in 
distance education see also Thorpe 1988, p. 7 or Holmberg 1995, ch. 10). In the following, 
we shall - without any claim to completeness - list up some of these tasks: 

• Market analysis: e.g. within the framework of course or programme planning: for instance 
assessment of the demand for a new offer of courses or programmes (e.g. questioning of 
employers on the desired qualifications of applicants as executed by the British OU). 

• Organizational evaluation: evaluation of internal arrangements and procedures; 
monitoring with regard to administrative processes and procedures; examples may be 
found in the article on the UNA in Venezuela by Machado & Machado or in Woodley's 
paper on the British OU. 

• Policy evaluation: evaluation of certain measures or regulations of the respective 
institution; e.g. examination of the effect of changes in fees on enrolment (see Woodley's 
paper). 

• Social needs analysis: diagnosis of the students' learning needs for the purpose of 
curriculum development and planning (see Murugan’s paper). 

o 
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• Student assessment/evaluation: monitoring and evaluation of the students' performance. 
Grading does often not merely serve the purpose of assesssing a given performance, but 
is also used in order to encourage and to motivate the students (see for instance 
Rekkedal's paper). The diagnosis of the students' previous knowledge at the stage of 
enrolment may be part of student assessment as well (entrance tests, see Murugan's 
paper). Moreover, all kinds of self-administered tests ‘whose sole purpose is to provide 
students with personal feedback’ (Potter) may be regarded as part of student 
assessment. 

• Course evaluation: For this purpose, different approaches and procedures are being 
used. In the sense of a formative evaluation or a 'developmental testing', evaluation may 
be carried out while drawing up course material (e.g. by 'critical commenting' or by 'peer- 
reviewing' executed by experts in didactics or in the relevant subject areas such as the 
‘initial piloting of the design with one expert’ at The Victoria University, testing of a course 
or parts of it with a rather small group of students before running this course); on the 
other hand, evaluation may also be carried out later, i.e at the stage of running the course 
- e.g. with regard to a scheduled revision of the course (e.g. by means of a questionnaire 
for students and tutors on their experience with that course). In their articles, Potter and 
Woodley will also go into the changes course evaluation has undergone in the course of 
time. 

• Media (including the "new media"): This aspect may range from the examiniation of the 
effectiveness of a given supplementary medium offer as part of a course package (e.g. a 
supplementary CD which is to illustrate a given course content) to the testing and the 
assessment of new methods of mediated communication (e.g. e-mail or computer 
conferencing), e.g. for communication between students and lecturers (for instance within 
a 'virtual seminar'), between students and tutors or for communication among students; a 
brief description of a corresponding project can be found in Rekkedal's paper. 

• Student services: The whole area of communication between students and the institution 
can be evaluated with regard to very different criteria and aspects. Tutorial support is one 
of the crucial elements of this area, and there are various evaluation tasks connected 
with this aspect: e.g. development of methods to select and to instruct tutors; 
development and testing of specific models of tutorial support; assessment of the tutors' 
activities in face-to-face sessions; or monitoring of the tutors' activities with regard to 
correcting and commenting of assignments or as regards the answering of the learners' 
questions. 

• System evaluation and ‘social impact’ analysis: This aspect does mainly concern the 
question of if and to what extent the institution in its entirety has been successful and 
efficient or achieved its aims. Among the topics dealt with under this aspect are for 
instance: 

• development of participants' number; 

• recruitment of certain target groups (if the institution's aim is e.g. to address 
specific target groups - for instance those groups which are at a disadvantage in 
the traditional education system -, then to what extent these groups are 
represented among those enroled may be examined); 

• rates of graduates or dropouts; 

• cost-effectiveness (e.g. in comparison to traditional educational institutions 2 ); 

• utilization of the materials of a given institution by other educational institutions; 



2 Of course, studies on cost-effectiveness may not only be referring to the insitution as a whole, but also to some of its 
sections, to a given course offer or programme or, for instance, to the media which are being used. 
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• recognition of certificates and diplomas by other institutions or by potential 
employers; acceptance of certificates by society; 

• career prospects for graduates on the labour market; influence of studies on the 
future professional career; acceptance of graduates by society; 

Given this variety of topics it is hardly surprising that many distance education institutions 
have set up special institutes or departments which have been given the task of carrying out 
evaluation. Moreover, this kind of evaluation carried out internally by such specialized 
institutes or departments may be supplemented by forms of external evaluation: This may be 
the case when a distance education institution itself attaches importance to being 
periodically assessed by external ‘neutral’ experts or when such external assessments have 
to be undertaken as a result of orders from controlling organizations or from those who 
finance the institution respectively. (See Rekkedal's article for a discussion of the pros and 
cons connected with internal and external evaluation.) 



Evaluation Methodology 3 

The variety of evaluation purposes and tasks (see above) corresponds to the diversity of the 

methodical approaches, procedures and methods that are used in evaluation, only some of 

which will be mentioned in the following: 

• qualitative approaches with a low degree of standardization often addressing only a small 
number of informants - as e.g. procedures of textual criticism (with regard to technical, 
didactic or linguistic criteria) for the purpose of course evaluation; ‘critical commenting’; 
‘peer reviewing’, 'piloting of the design' with experts; 

• rating procedures (e.g. student assessment or the evaluation of the didactic quality or 
effectiveness of course components by experts or by students); 

• knowledge or retention tests (e.g. for the purpose of student assesment or in order to 
examine the effectiveness of a course); 

• survey methods and questionnaires: semi-structured or standardized interviewing of a 
larger number of informants (mostly by means of written questionnaires; sometimes also 
by telephone interviews); 

• face-to-face interviews, including in-depth interviewing (e.g. in order to gain an insight into 
how learners deal with the subject matter); 

• methods of observation (e.g. as regards the assessing of the tutors' behaviour and 
performance in face-to-face sessions); 

• statistical analysis of available data (e.g. analysis of enrolment data in order to examine 
representation of different groups among students); 

• methods of content analysis (e.g. with regard to course contents while evaluating a 
course or as regards communication between learners and tutors as a part of an 
evaluation of student services); 

• cost-effectiveness or cost-benefit analysis (e.g. in order to compare different teaching 
methods or combinations of media); 

• experimental or quasi-experimental studies (e.g. in order to estimate the effectiveness of 
certain kinds of tutorial support; cf. for instance Rekkedal's article); 
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There are several arguments in favour of the utilization of a variety of methodological 
approaches and methods: 

In this respect, for instance McGrath (1982) elaborated the incompatibility of the three 
general quality aspects of research in the social sciences (i.e. ‘generality’, ‘precision’, and 
‘realism’). He points out that by every choice of a distinct research strategy (e.g. a laboratory 
experiment or a specified form of field research) or inquiry method (e.g. a specified form of 
interviewing or observation), one cannot simultaneously achieve an optimum with regard to 
all of the three aims or quality aspects. One is either only able to attain an optimum with 
regard to one of the aims at the expense of the two others or to achieve a relative optimum 
as regards two of the aims while giving up the third (‘dilemmatics’). 

Cronbach (1982), too, opposes conceiving certain kinds of methodological approaches and 
study designs in the area of evaluation as universally valid ideals while neglecting the 
utilization context (e.g. the concrete demand for information which stimulates an evaluation 
study). Consequently, he demands a functional way of looking at the problem: "A design is a 
plan for allocating investigative resources [...] In evaluation - as in basic science - the 
designer’s task is to produce maximally useful evidence within a specified budget of dollars, 
a specified number of person-years from the evaluation staff and of person-hours from 
informants, and other such constraints. ‘Maximally useful’ is a key phrase. Most writings on 
design suggest that an investigation is to be judged by its form, and certain forms are held 
up as universal ideals. In contrast, I would argue that investigations have functions and that 
a form highly suitable for one investigation would not be appropriate for the next. This is to 
be, then, a functional theory of design." (Cronbach 1982, pp. 1-2). 

In a similar way, Patton (1987, p. 169), too, demands that evaluators have to be flexible as 
regards their choice of methods. Ideally, they should be able to adapt their choice of 
research methods to the specific purpose of the evaluation, to the peculiarities of the 
evaluation object and to the information needs of the decision-makers: "[...] evaluators must 
have a large repertoire of research methods and techniques available to use on a variety of 
problems. Thus today’s evaluators may be called on to use any and all social science 
research techniques, social indicators, cost-benefit analysis, standardized tests, 
experimental or quasi-experimental designs, unobtrusive measures, participant observation, 
and depth interviewing. “ (Patton 1987, p. 169). 

Besides methodological criteria, decisions on methods in evaluation thus have always to 
consider the context in which the evaluation is carried out (e.g. the time or budget available) 
and, particularly, the purpose of evaluation and the utilization context. 

* * * 



Finally, we would like to thank the authors of the papers. We are greatly obliged to them for 
their willingness to cooperate. 



Harish Rathore and Rudolf Schuemer 
Hagen, February 1998 
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Krishnapillai Murugan 
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Maidan Gar hi, New Delhi - 1 10 068 

India 



1 Background 

As per item 66 in the Union List of the Seventh Schedule in the Indian Constitution, 
‘coordination and determination of standards in institutions for higher education and research’ 
is the responsibility of the Central Government. The University Grants Commission (UGC, 
for short) thus was established as a statutory body in 1956 to perform this function. „It shall 
be the general duty of the Commission to take, in consultation with the universities or other 
bodies concerned, such steps as it may think fit in the promotion and coordination of 
university education and for the determination and maintenance of the standards of teaching, 
examination and research in universities...." (Section 12 of the UGC Act, 1956) 

The establishment of the Indira Gandhi National Open University (IGNOU) in September, 
1985 by an Act of the Indian Parliament brought about a major change in this situation. 
Sections 4 and 5(2) of the IGNOU Act specify, inter alia, that „The objects of the University 
shall be to advance and disseminate learning and knowledge by a diversity of means [...] to 
encourage the Open University and distance systems in the educational pattern of the country 
and to coordinate and determine the standards in such systems...." Thus, the university has a 
unique mandate of functioning both as 

• an open university like any other in the rest of the world, i.e. generation and dissemination 
of knowledge through the distance mode, and 

• an apex body to co-ordinate and maintain the standards of distance education systems in 
India. (As of now, besides 7 state- /province-level open universities, a little more than 50 
out of about 230 on-campus universities have distance education units). 

To fulfill its role as a university, IGNOU functions in a three-tier system of Headquarters, 
Regional Centres (#17) and Study Centres (#268). While the ‘manufacturing’ aspect of 
distance education such as course production in a mass scale, etc., is being looked after 
predominantly by the staff at the Headquarters comprising academic Schools (#10) and 
support Divisions (#10), (which are headed by Directors/Registrars working under the overall 
supervision of three Pro- Vice-Chancellors), the ‘servicing’ aspect is the main concern of that 
at the Regional and Study Centres. The Vice-Chancellor is the Chief Executive Officer of the 
university. IGNOU is governed by such statutory bodies as Board of Management, Academic 
Council and Planning Board. The President of India is the Visitor of the university, which 
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comes directly under the Union Ministry of Human Resources Development unlike other 
universities in India which are governed by the UGC. 

Depending on the area and the student strength, each Regional Centre looks after 
approximately 1 6 Study Centres, equipped with a library, audio/video aids and the provision 
for face-to-face interaction. Barring a few cases, the rest of the Study Centres have been 
housed in conventional educational institutions. Face-to-face counselling sessions and 
practicals, for courses which need them, are conducted in these Centres. All the Regional 
Centres and some of the Study Centres also have the satellite downlink facilities for purposes 
of teleconferencing. 

The Regional Centres, while co-ordinating the functions of the Study Centres that come 
under their respective purview, provide an interface between them and the Headquarters. Each 
Regional Centre is headed by a Regional Director assisted by 3-4 Assistant Regional 
Directors whose task involves conducting training programmes for the academic-counsellors 
(#16,364) with the academic support from the Headquarters; visiting Study Centres for 
purposes of monitoring their student-related activities, etc. Besides, the university has 
established three functionally autonomous units with specific objectives to carry out. Thus the 
Distance Education Council (DEC) takes the second of the two roles of IGNOU; the Staff 
Training and Research Institute of Distance Education (STRIDE), which has been conferred 
with the title Centre of Excellence for Training in Distance Education, by the Commonwealth 
of Learning (CoL), helps in system building within India and without and the Educational 
Media Production Centre (EMPC) is a national resource centre for educational media. 

With right earnest, the first courses of the university were launched in January 1987. 
Currently, it offers 482 courses (in #74 programmes) on a variety of disciplines ranging from 
counselling, child care, computer, etc., to engineering, environment, distance education, 
health sciences, etc. The policy of the university is that it should strive to offer academic 
courses/programmes which are student-driven and thus are distinguishable from the typical 
programmes available in the formal educational set up. Put differently, the university takes 
conscious efforts to provide socially relevant education, moving away from discipline-centred 
curricula to learner-centred ones. The mission statement of the university that it should reach 
people and places hitherto unreached makes it amply clear that IGNOU takes concerted 
efforts to identify new and unconventional clientele groups. Presently, the cumulative student- 
strength is a little less than 400 thousand. 

This background, though a little lengthy, is essential to understand the evaluation practices 
at IGNOU, as evaluation is just one sub-system of a larger system, i.e. the university, in the 
present context. 

As IGNOU has to evolve a model of its own, in the absence of one to fall back on, ad 
hocic approach to various system-related activities, which include the sub-system of 
evaluation, is still prevalent at the university. However, with the introduction and extensive 
use of computers for purposes of evaluation, this sub-system is gradually getting streamlined. 



2 Evaluation as perceived at IGNOU 

The university policy recognizes, broadly, the following seven types of evaluation, depending 
on the context in which the term is used: 

1 . assessing social situations and the learning needs of learners for purposes of curriculum 
design/development (social needs analysis). 
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2. diagnosis of the entry competence level of learners for admission into various 
programmes/courses (i.e. entrance tests). 

3. continuous (formative) and terminal (summative) evaluation of student performance for 
certification (i.e. student assessment). 

4. evaluation of academic/training programmes and related activities for feedback and 
improvement (programme evaluation). 

5. evaluation of the academic courses/programmes of state- /province-level open universities 
and distance education systems attached to the formal educational institutions for purposes 
of national accreditation, pooling together of resources and avoiding duplicatory efforts. 

6. evaluation of the functions/activities of the distance education systems in the country for 
providing financial assistance, etc. 

7. evaluation of the impact of IGNOU courses/programmes on the society and their 
contribution for national development. 

Excepting the last type in the list above, the rest are either fully or partially operational. 
Plans are underway to carry out the evaluation of the impact of the courses/programmes of 
IGNOU on the society. Evaluation of academic courses for accreditation and evaluation of the 
functions of distance education institutions for funding purposes come under the role of 
IGNOU as an apex body for distance education in India. (The Distance Education Council, as 
referred to earlier, performs this function). 



3 Evaluation for programme design/development 

In India, conventional universities could cater to just 6% of the aspirants for higher education, 
though there are about 230 such institutions. And, the demand for higher education is ever- 
increasing. It is against this backdrop that the establishment of the Indira Gandhi National 
Open University should be seen. The two main academic reasons, which can be attributed to 
its establishment, thus, are 

• to make available higher education to a wider segment of the adult population, which 
seeks it, and 

• to launch ‘socially relevant’ courses/programmes. (‘Knowledge explosion’ warrants that 
innovative, market-oriented courses/curricula are made available to students, preparing 
them for nation-building in the future). 

Conventional universities, though make every effort to overhaul the curricula, are 
structurally and functionally inadequate to cope with the speed demanded of them to carry out 
this task and the phenomenal number of aspirants to be educated in a short period, owing to, 
among other things, their inherent academic restrictions of various kinds. Contrastingly, the 
inherent flexibility of the open systems helps IGNOU satisfy these demands. This being the 
case, ‘social needs analysis’ assumes an important role to play in the programme development 
process at IGNOU. Though generalistic type conventional courses/programmes may not 
require needs analysis, for courses that are innovative and specific to a particular target 
clientele, market surveys/researches are conducted to ascertain the need for launching such 
courses/programmes. Once the university is convinced of the need, on the basis of the 
feedback on the market survey (normally conducted by a team of people). Schools concerned 

1 ° 
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develop a draft curriculum, which will be discussed at a meeting of experts. Besides carrying 
out suitable modifications, this meeting helps identify some course writers from outside 
IGNOU as well - the idea is to make the materials pan-Indian. The revised curriculum-outline 
will later be sent to the identified course writers for writing the courses. Those who agree to 
associate with IGNOU in the course production processes are given a two-day orientation 
programme in developing self-learning materials. Once the manuscripts reach the faculty 
concerned, some of which also contribute to writing, they will be edited and camera ready 
copies will be sent for printing. Thus, for programme development, market surveys/researches 
are conducted from time to time. It should also be noted that not all the courses/programmes 
available currently are bom out of the results of market surveys. But, it is not in the distant 
future that IGNOU has to undertake feasibility and market demand surveys for justifying the 
existence of all the courses/programmes. It is obvious that this initial evaluation, i.e. market 
surveys/researches, leads to the development of new curricula and, therefore, this exercise is 
seen essentially as an exercise in programme development. 



4 Evaluation of student entry 

Evaluation of student entry here refers to entrance tests conducted by the university under 
study for purposes of diagnosing the student competence level and/or student-aptitude. 

To demonstrate the flexibility of and openness in the open educational systems, IGNOU 
offers programmes/courses with relaxed entry qualifications - entry qualifications which do 
not match with what they are at conventional universities. However, the university prescribes 
a few other entry competences or skills. For example, the students who do not have the 
requisite conventional entrance qualifications for admission to first degree programmes, such 
as B.A., B.Com and B.Sc., are allowed to join them provided 

• they have attained 1 8 years of age at the time of admission, and 

• they qualify at an entrance test specially designed for the purpose. Normally, it purports to 
test a prospective student’s general awareness about the environment he/she lives in, 
his/her language competence and his/her skills at numerals and analytical abilities. 

• they undergo what is called a ‘Bachelors’ Preparatory Programme’ of a certain duration, 
which is something akin to a bridge course, to prepare the students for the regular 
programme. Students are evaluated on the basis of assignments, the grades of which will 
not form part of the overall grade a student will get at the end of the first degree 
programme. Further, success in this programme does not entitle the student for any 
certification but it facilitates students to get into the regular programme. 

Currently, the university conducts such tests as mentioned for the management and 
computer-related programmes, for testing essentially the aptitude of the prospective students. 
Besides, on the basis of a self-appraisal form, which the students are required to fill in, 
students get admission into a programme in Creative Writing. With the passage of time, the 
number, variety and format of such tests are expected to increase, as more non-conventional 
courses are underway that allow open entry. 

It should be made clear here that success at the entrance tests does not qualify a candidate 
for any type of certification, instead achieving a predetermined minimum norm at such a test 
entitles the candidate to admission to the programme for which the test is conducted. 
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5 Evaluation of student performance 

However desirable and imperative, breaking new grounds in an educational system as a whole 
or in any one of its sub-systems have essentially been a process rather than an event. We are 
aware, for instance, that there have been discernible changes in the practice of educational 
evaluation from time to time. However, we cannot assign any specific time frame practising 
one particular pattern at a given period. It is therefore possible that a cluster of patterns 
coexists. What appears to be defunct at a particular context need not necessarily be so at a 
different situation. In essence, depending on relative merits either one pattern of evaluation or 
a combination of a few patterns gains currency. And this exactly is the situation at IGNOU. 

As regards evaluation of student performance, two assessment patterns are in vogue in the 
university: 

• continuous assessment of student-performance 

• end of the term assessment of student-performance 

These assessment patterns are not unfamiliar to us in the context of classroom 
teaching/leaming process. Notwithstanding, they deserve to be discussed in the distance 
education contexts with particular reference to the Indira Gandhi National Open University. 
This is so primarily because there is a shift in emphasis in the purposes for which they are 
employed, especially in the case of continuous assessment. 



5.1 Continuous assessment (Formative) 

Traditionally, continuous assessment has been and still is used as a means through which the 
academic performance of a student is measured. At IGNOU, continuous assessment is done 
through tutor-marked assignments (TMAs) and computer-marked assignments (CMAs). 
However, besides grading the academic performance of a student, in present context, 
assignments are being used to monitor the whole learning process of the students. 

Thus the functions of an assignment at the IGNOU context are: 

• to assess the academic achievement of a student 

• to facilitate long term and short term information retention 

• to provide feedback (to students/institution) 

• to sustain motivation and break isolation 

• to establish two-way communication 

In essence, assignments are used for distance teaching/leaming purposes. This being the 
case, considerable attention is paid in the preparation of assignments, computation of grades 
awarded and the turn-round time. 

Basically, the objectives of a course/programme and the nature of the discipline determine 
the assignment questions - whether they are tutor-marked or computer-marked. A checklist of 
the following type is used for the design of an assignment question: 

• Will each learner interpret the question in the way it is intended to be read? 
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• Does the question come under the purview of the learning objectives formulated and the 
course content identified? 

• Are there several ways of answering the question adequately? (It is possible. However, it 
does mean that alternative marking schemes may be needed for a variety of good answers. 
In other words, it is a good idea not to make the question too open so that there is only 
really one best way of interpreting and tackling it.) 

• Does it prepare the learner for the sort of questions that may be met in the term end 
examinations? 

• What is the main purpose of the question - assessment or feedback to the 
leamer/institution? (In the distance education context, both should be served through an 
assignment question. However, it is likely that certain questions can be so designed to get 
the feedback from the students rather than to give feedback to them or the vice versa.) 

• Should model answers be sent to learners? Are there any criteria for this purpose? 

• Does the assignment as a whole test the main things the learner needs? (It is too easy for 
an assignment to test a particular cross-section of what the learner needs to know because 
that cross-section happens to be easier to test.) 

• Does the assignment follow the easy to complex philosophy, i.e. beginning with fairly 
easy questions, say for example, reference-oriented and moving towards something which 
is relatively complex, say application-oriented? (The most difficult for any learner starting 
to do an assignment is to pick up a pen and make the very first mark on the page. Once the 
learner is prepared, progress usually continues much more smoothly. An easy first 
question can instil enough confidence into the learner to get going.) 

• Does the assignment provide for appropriate either/or choices? (This can help the learning 
materials serve a variety of learners with different needs, abilities, interests and 
experiences. However, marking/grading scheme has to be rationalised.) 

The list is by no means exhaustive. From time to time, IGNOU adds to it. 

Barring a few occasions, academic counsellors themselves evaluate the assignment 

responses of the students. The university gains insights from the comments of these academic 

counsellors who react to the assignment questions on the basis of the checklist presented here. 

Assessment of the assignments and turn-around time are the important issues as well. The 

general practice in use at IGNOU is given in 5. 1 . 1 . 



5.1.1 Operational scheme 

Assignment questions are prepared by the Schools concerned at the Headquarters, which are 
sent to students by the Materials Publication and Distribution Division (MPDD). Normally, a 
4 credit course (one credit is equivalent to 30 hours of student-input) has 2-3 TMAs and 1 
CMA - though this proportion is not mandatory. As regards TMAs, the students send their 
responses to the assignment questions to the coordinators of the Study Centres, to which they 
are attached. They, in turn, distribute the responses to the counsellors identified for the 
programme/course. After evaluation, the counsellors send them to the Study Centres for 
onward transmission to the students. And, copies of the required percentage, i.e. 2% to 5%, of 
the evaluated responses are usually sent to the Schools concerned for purposes of monitoring 
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the performance level of the counsellors, among others. (Responses to CMAs are sent directly 
to the Headquarters). 

The whole process can schematically be represented as under: 



2% to 5% copies of responses 



As. Qns. As. Res. 

IGNOU *• Learner *■ Study Centre 



Graded responses 



Counsellors 

Fig. 1: Assignment evaluation process 

The university, therefore, evaluates the whole process to find answers to such questions 
as: 

• Do the assignment-questions reach the students in time - assuming course materials reach 
them in the scheduled time? (Corresponding issues could be to find out the reasons, if the 
assignment questions do not reach the students in time and suggest possible solutions to 
‘alleviate’ the problems, if any.) 

• Does the university give sufficient time for the students to prepare the responses? 
(Correspondingly, the university should see whether the students adhere to the stipulated 
time, assuming they get sufficient time for preparation. And if the university extends the 
time-period, it should state the rationale and possible administrative implications.) 

• When does a co-ordinator send the responses that he/she receives to the counsellors for 
evaluation? (In correspondence to this, the university sees whether or not any norm is in 
vogue; whether the counsellors are given enough time for evaluation and whether the 
counsellors adhere to the stipulated time-frame. Supposing the counsellors are prompt, 
how much time does a Study Centre require to despatch the responses back to the 
students?) 

• What are the general constraints and how might they be ironed out? 

The items listed here indicate that they are mostly administrative in nature and the bottlenecks 
are being gradually removed. 

As regards the academic aspect, i.e. that of monitoring the evaluation of assignment- 
responses, IGNOU lays stress on 

• awarding a legitimate grade to a response 

• writing comments on the responses to facilitate learning 
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5.1.2 Awarding grades to responses 

At IGNOU, this process entails: 

• thoroughly studying and assessing course-units and assignment questions 

• understanding what the assignment demands of the learner 

• preparing an ‘ideal response’ to the problem posed in the assignment 

(An evaluator is not required to write the whole answer. However, he/she can note down 
the points with regard to information input and organization that should constitute a 
response. This helps award appropriate grades to responses.) 

• identifying such of the weaknesses of the responses as might be traced back to the defects 
of the assignment-questions and/or the course unit. 

(This has a great pedagogic significance. If he/she is carried away by his/her ‘ideal 
response’ irrespective of the defects of the materials, the assessment is likely to be 
lopsided.) 

It is, however, possible that one is carried away either by information density or by the 
organisational style. It is therefore essential to assign fairly precise values to various aspects 
of a response. Moreover, this practice helps in reducing discrepancies in grading. 



5.1.3 Writing comments on responses 

Most distance students enrolled at IGNOU are new to the distance mode of teaching/leaming 
process. And so, the university takes every effort to initiate them to distance education. One 
such effort is to orient the academic counsellors to write tutor-comments on the assignment- 
responses of the students. Unlike a classroom teacher who mostly addresses a collective mass 
of students, the university strives to ensure that every student is addressed to individually. 

Tutor-comments, at the IGNOU, therefore, subsume 

• correcting the students and guide them where they may have gone wrong, 

• elaborating on what the students may have attempted summarily, 

• pointing out and confirming the acceptable aspects of the responses, 

• explaining the basis of the assessment. 

This is done at two levels. Academic-counsellors are required to write specific comments 
relating to particular aspects of the assignment-responses. In the IGNOU parlance, these 
comments are called ‘local comments’ or ‘marginal comments’ - comments written on the 
margin. A consolidation of these comments helps an academic-counsellor to award a suitable 
grade. However, for purposes of providing the justification for the grade awarded and of 
making overall comments, IGNOU uses ‘global comment sheets’ wherein an academic- 
counsellor makes comments regarding the overall merit of the responses specifying such 
matters as language, presentation, clarity, comprehensibility, etc. 

The basic reason for this practice is that most of the time during the teaching/leaming 
process, the students are away from the institution/teachers. A counsellor at IGNOU is, 
therefore, to find occasions to show the students that he/she is the most sincere confidant(e) 
whose prime aim is to assist the learners in their studies. Tutor-comments are being used for 
this purpose. His/her written comments must also compensate for the mannerisms which 
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accompany and take away the sting from the teachers’ remarks in a classroom situation. The 
comments, therefore, will have to be well-thought out, deliberate, palatable, precise and 
pedagogically purposeful. Obviously, writing comments makes an IGNOU counsellor’s task 
more difficult and challenging than that of a classroom teacher. 

However, the university under consideration has not devised a system by which the 
counsellors will come into contact periodically with the co-ordinators and/or the Regional 
Centres. At the initial stages, perhaps, the need was not felt and also the infrastructure of the 
Regional Centres was not adequately established. It is time the university took note of this 
issue. Besides, the university should try to involve the co-ordinators of the Study Centres in a 
more extensive way in the management of the turn-around time of assignment-responses. 



5.2 Terminal examination (Summative) 

At present, IGNOU conducts terminal examination, popularly referred to as term-end- 
examination, twice a year - normally one in June and the other in December of a particular 
year. Almost all of the IGNOU programmes are calender-year programmes, i.e. programmes 
commence in the month of January. As there is flexibility in the duration of completion of a 
programme, the conduct of the terminal examinations twice a year facilitates student pacing. 
For example, a student of the Postgraduate Diploma in Distance Education can complete the 
programme comprising 5 courses either in a year or within a maximum period of four years, 
depending on his/her disposition, convenience, etc. Suppose that this student has enrolled for 
the programme in January 1997. He/She can appear for the terminal examination for the first 
time in December 1997. If he/she could not take examinations for all the courses, he/she can 
attempt the examinations of June/December of the subsequent years till December 2000. No 
extra fee is charged for multiple appearance for the examinations. Put differently, the 
programme fee includes examination fee as well and fee once paid is valid for four years in 
this case. (And, depending on the programme, the duration varies - for instance, a first degree 
student can complete the programme in a minimum period of 3 years or in a maximum period 
of 8 years). For appearing for the examinations, the students have to intimate to the Student 
Registration and Evaluation Division (SR&ED) indicating the number of course(s) they wish 
to appear for in March and September of a year respectively for June and December 
examination. 

At IGNOU, it is customary to use a 3-hour written examination for purposes of terminal 
evaluation. Usually, each question paper has 3 sections, one each for an essay type question, a 
few short answer type questions and an application-oriented question sequenced in that order. 
And, no multiple-choice item figures in the term-end-examination. Examinations are 
conducted in all the Study Centres of the university across the country. (For an international 
programme, i.e. Master of Arts in Distance Education (MADE), examinations are conducted 
in as many as 1 5 countries in the Commonwealth). 

Two sets of question papers for the terminal examination are generated both by the faculty 
at the School concerned and by some external experts in the relevant field. To maintain 
confidentiality of the whole exercise, the first drafts are hand-written. The SR&ED plays a 
crucial role in this activity. After collection of the required number of question papers, the 
SR&ED in consultation with the Schools concerned constitutes a ‘moderation committee’ in 
which services of at least one expert from outside the university (besides the faculty 
concerned and the officers of the SR&ED) are used to fine-tune the question papers, i.e. 
removing ambiguity in the questions, maintaining the syllabus coverage, etc. 
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The university under discussion is now planning to establish on-demand examination 
scheme (as a consequence to its decision to introduce walk-in admission scheme) wherein a 
student can take the examination at any time depending on his/her convenience. Though at the 
policy level it has been accepted, the implementation of this would take a few years from 
now. However, as an initial gesture to facilitate this scheme, question-banks are being 
established. 

It should be noted here that submission of a required number of assignment-responses is a 
prerequisite for taking the examinations. 



5.3 Evaluation scheme 

As mentioned, for purposes of continuous assessment, IGNOU uses assignments and for those 
of term end assessment, examinations which are conducted twice in a year to facilitate 
student-pacing. Usually, the weight given to the former is between 25% - 30% and that to the 
latter is between 70% - 75% of the total. The submission-rate of assignments, more often than 
not, is determined and rationalised on the basis of the nature of course-demands. 

The Indira Gandhi National Open University follows a five-point grading system as given 
below: 



Point 


Letter Grade 


Notional Correlates 


Grade point 


5 


A 


Excellent 


4.5 and above 


4 


B 


V. Good 


3.5-4.49 


3 


C 


Good 


2.5-3.49 


2 


D 


Satisfactory 


1.5-2.49 


1 


E 


Poor 


1 .49 and below 



Table 1 : Grading scheme at IGNOU 

A student at IGNOU thus gets an overall grade for a course by completing the required 
number of assignments and fulfilling the requirements of the examination. Usually, a 
minimum grade is prescribed for successfully completing a course. However, it depends on 
the programme demands. An example of the computation for final grading, given that the 
weight allotted to continuous assessment and term-end examination respectively is 25% and 
75%, is presented below: 







Grade 


Point 


Average 


Course X 


Assignment 1 


B 


4 


1.12 




Assignment 2 


A 


5 






Examination 


B 


4 


3.00 



The total of the average worked out (i.e. 4.12) is the grade point scored by a student. If 
plotted against the grade range, one arrives at the grade to be awarded (i.e. B) to a student. If 
Course X mentioned above has a CMA and/or one more TMA the computation will be done 
accordingly. 

At IGNOU, as mentioned, a flexible programme duration is being followed. For instance, 
one can complete any first degree programme either in 3 years or in a maximum period of 8 
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years and a diploma programme in one year or 4 years, etc. Put differently, once the students 
pay a one-time course/programme fee, she or he will be on the IGNOU rolls till she or he 
successfully completes the course/programme or the maximum period or drops out, as the 
case may be. Importantly, if a student is able to complete only a few assignments and/or the 
examination of a few courses, he or she can carry forward the grades till the stipulated 
maximum period. However, assignment questions are changed every year. (There is a 
provision for revaluation of the examination papers on demand and on payment of a particular 
fee, if a student so desires.) 



5.4 Implications for orientation of academic-counsellors 

In the absence of any model, initially Regional Centres were set up with a view to co- 
ordinating the administrative activities of the Study Centres and reporting to the Headquarters. 
And, when the idea of decentralisation was floated and then partly implemented, some 
academic infrastructure was created, though uniformity in terms of number in the appointment 
of academics could not be maintained. Of late, depending on the expertise available. Regional 
Centres are being involved in orienting the counsellors for purposes of assignment evaluation 
(and face-to-face counselling). Unfortunately, however, once the counsellors take up the task 
of assignment evaluation, the whole activity is restricted within the ambit of the Study 
Centres. Little efforts, however, seem to have been initiated in evaluating the orientation 
programmes and/or the effectiveness of the academic-counsellors. 

Put differently, notwithstanding the fact that initial orientation programmes are 
generalistic in nature, it is highly essential to get the feedback from the counsellors about the 
training imparted and how it helps them, if at all it does, to carry out their intended tasks. One 
crucial area for feedback is the evaluation of assignment-responses. But, as of now. Regional 
Centres do not have any role to play in the whole process of evaluation. It is very evident from 
the schemeta presented earlier (see Figure 1) . The implication is that to monitor this process 
and in essence to develop the required human resources, the Regional Centres need to be 
involved. Keeping this in view the university should restructure the schemata as under: 



As. Qns. As. Res. 

IGNOU ► Learner ► Study Centre 



I Evaluated Res. * * 

Regional Centres Counsellors 

I I 

Feedback 

Fig. 2: Assignment evaluation process: Restructure 

In the proposed schemata, the management of the evaluation of the assignment-responses 
will entail: 
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• the Study Centres informing the Regional Centres concerned as to how many assignment- 
responses have been sent to whom and when for evaluation. 

• the Regional Centres checking the number of responses returned by whom and when. 

• the Regional Centres going through the evaluated responses at random and sending them 
to the students with the global comment sheets. 

• the Study /Regional Centres sending the copies of the required percentage of evaluated 
responses to the Schools concerned as a feedback on their training/materials. 

• the Study/Regional Centres or the Schools concerned giving feedback to the counsellors. 

Interaction between the Regional Centres and the counsellors will help establish a useful 
academic rapport which in turn may help instil in the counsellors a sense of accountability and 
belongingness. Considering the present structure of Regional Centres, and the nature of 
responsibility they have been invested with, they may not be able to take up these tasks. 
However, they need to be involved though in gradual phases. A beginning, therefore, can be 
made with required modifications in the above schemata. The emerging alternative model 
may be presented as under: 



Graded Responses 



As. Qns. 



IGNOU 



As. Resp. 



Learner 




Study Centre 



Counsellor 



Fig. 3: Assignment evaluation: An alternative structure 

However, before this activity begins, it is impending on the university to arrive at a 
consensus as to what exactly the role of the Regional Centres is regarding the management or 
monitoring of responses. This is important because for obvious reasons expertise for all the 
disciplines/subjects will not be available in a Regional Centre and, therefore. Regional 
Centres cannot look at either the content validity of the responses or those comments of the 
counsellors which have specific reference to the content. By implication, all that the faculty at 
the Regional Centres can look into is the way comments have been written, i.e. from the point 
of view of distance education. 

As a pedagogically significant measure, the Schools concerned that prepare the 
assignments should evolve guidelines for evaluation. This includes such suggestions as what a 
particular assignment demands, what kind of responses would deserve what grades, etc., and 
send them to the Regional Centres which in turn send them to the counsellors through the 
Study Centres. Despite some logistical problems, the materials have to be sent through the 
Regional Centres so as to involve them in the process of assignment-monitoring. 
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6 Programme evaluation 

At IGNOU, the term ‘programme evaluation’ is used to mean an evaluation of an academic 
programme as a whole or a part thereof or a training imparted or service rendered. It also 
includes both print and electronic media. To appreciate this exercise, it is essential to acquaint 
ourselves with the instructional system of IGNOU. 



6.1 Instructional system 

The master medium used at IGNOU is the printed word in the form of self-learning materials. 
An academic programme consists of a few courses; each course is divided into a few blocks 
and a few units - normally 4 units of 20-25 pages each constitute a block. Materials are 
printed in blocks. Usually, less than 20% of study time is to be devoted to audio and video 
programmes and face-to-face academic counselling sessions. Besides these, some 
programmes have additional components in the form of practical work in the laboratories, 
field studies, case studies, hands-on experience of computers, etc. (As a value-added service, 
of late, the university uses teleconferencing). Each course of IGNOU is worth a few credits 
and each credit is equivalent to 30 hours of student- input, which includes reading the 
materials, writing the assignment-responses, watching the video programmes, listening to 
audio programmes, attending face-to-face academic counselling sessions, etc. As electronic 
media are used as a supplement to the print materials, it is not mandatory on the part of the 
students to use them and the same is applicable to attendance in counselling sessions, except 
in the case of the Postgraduate Diploma in Higher Education and teleconferencing sessions. 

The university under discussion prescribes a course-life of normally 5-7 years to every 
course on offer. During this period, the learning materials are updated from time to time and 
at the end of the stipulated course-life, the courses are completely revamped, primarily on the 
basis of feedback and depending on the socio-academic changes. For feedback, the university 
depends on various sources which include expert opinion, market demands, reactions of the 
students, academic counsellors, the enlightened public and other stakeholders. 



6.2 Evaluation of printed coursewares 



As Koul (1991) points out, at IGNOU, programme evaluation of academic programmes 
comprises two components - evaluation of the very planning of a programme/course and 
evaluation of materials that result from the plan. The activity involved in the former is to 
evaluate a programme/course in terms of the need for it, its marketability, its utility, its 
economic viability, adequacy and appropriacy of its content and media components. As an 
illustration, Koul (Ibid) elaborates on one of the above factors - the economic viability of a 
programme/course. For this, various criteria are used to indicate as to how much is to be spent 
on preparing a course and what might be the returns. In particular, attempts are made to see as 
to how much money needs to be spent on the various operations pertaining to a particular 
course such as cost of expert-committee meetings, orientation programmes for course writers, 
paper and printing, audio and video materials, support services and the equipment needed for 
these services, and the cost of learning as far as the student is concerned. These cost-factors 
help in estimating whether a course is cheaper than the other, etc., and thus establish their 
relative cost-effectiveness. 
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However, it should be mentioned here that IGNOU is still at the experimental level as 
regards evaluation of curriculum planning. Nevertheless, attempts are being made to develop 
this type of evaluation. The idea is to evolve guiding principles to be followed in the future 
when the resources for course production and implementation will be relatively more difficult 
to find and they will need to be managed with great care to save the university from possible 
surprises and difficulties. 

For evaluation of the materials produced, a checklist of the following type is generally 
used: 

• is the course content adequate? 

• is the language used appropriate to the level concerned? 

• is the material adequately self-instructional? 

• is the media utilisation pedagogically rational? 

• is the material easily accessible? 

• is the course material considered useful by academics outside the university and are they 
using it in their systems? 

• does the material conform to the institutional norms of format, size, etc., as required by 
the university? 

• does the material help the student to get a good pass in the examination? 

This discussion should suggest that programme evaluation is a two-tier operation - the 
first consisting of those elements of evaluation which are a part of the general management 
process. For example, the rationale behind and the outline of a particular programme/course is 
looked into by an expert committee, modifications brought in at this stage are again discussed 
and reviewed with the course writers; in the process of course writing the materials go through 
the hands of a language editor, a content editor, a format editor and a chief editor, etc., 
depending on the need, and at the post-production level informal/unsystematic feedback is 
obtained from students, counsellors and others who matter. All these stages are the 
components of programme/course evaluation as a part of the management process - the 
establishment has not to put in any separate resources for this level of programme/course 
evaluation. However, the second tier of course evaluation may or may not fall within the 
general management process. It may have anyone or more of the following forms: 

i) piloting of programmes/courses: In this scheme the materials are tried out with the first 
batch of students and revisions brought in subsequently. This is a well-known approach and 
very often the management takes to it almost without giving any thought to its utility in 
relation to its cost. The following are strong reasons why IGNOU should not depend on this 
approach for each and every course: 

• given the constraints on the resources (financial as well as human), it will not be possible 
for most courses to follow this approach and then offer an improved course to the second 
batch of students. 

• it is also not within educational ethics to use the first batch of students as guinea pigs 
always and ever. 

• correspondingly, the credibility of a course will be in question, should different batches of 
students get different kinds of courses and yet be assessed by the same examination 
procedures. 
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• operationally, bringing in revisions immediately after the first launch adversely affects 
distribution and support services. 

• the cost of course design will be prohibitive and the rate of course production also get 
adversely affected by this approach. 

• if the first version of each and every course is prepared hastily, the image of the university 
will be affected adversely in the long run. 

It is not, however, suggested that piloting a programme/course is entirely a fruitless task. 
Better ways of achieving similar results, nevertheless, should be explored. For example, the 
concept of ‘quality assurance’ may be inculcated in the production processes. 

ii) routine evaluation: This is best done as a part of management process. The advantage 
of keeping this evaluation within the management process is that in so doing it will not 
demand high resource allocations. Besides, it will keep the institution in constant touch with 
the students and the materials, resulting in appropriate and timely feedback to the academic- 
schools, course-writers and academic-counsellors. It will eventually feed into the process of 
course maintenance which can be carried out through supplementary materials and thereby 
saving the resources. Routine evaluation, thus, implies 

• evaluating each course within the year it is launched, and 

• evaluating assignment-questions and the term-end-examination question papers, in 
addition to course materials. 

If the above two raise issues that cannot be explained easily and convincingly, special 
evaluation is conducted. 

Hi) special evaluation: This should come in as a project, whenever needed and proposed 
and for which the resources are requested for or they are available from a source other than the 
university. Obviously, such evaluation will need to have a purpose and, therefore, resources 
should be available for fulfilling that purpose. Thus, special evaluation of a programme/course 
needs to be considered on the basis of two factors: 

• the evaluation is for a special purpose, 

• there are resources available for this special evaluation. 

Such evaluation is obviously not a matter of routine. It has to be selective. Special 
evaluation is necessary in cases 

• of unusual and inexplicable poor performance of students or that of a programme/course, 
including adverse reactions by students, academics and counsellors; 

• where a new course design is introduced, a radically new target group is catered to or quite 
an innovative course is launched; 

• when a funding authority or an external agency asks for such evaluation with regard to a 
particular programme/course; 

• of programmes/courses which involve huge investments (in terms of financial and human 
resources) and/or take large numbers of students. 
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6.3 Evaluation of media materials 

IGNOU has, over the past 10 years, produced nearly #700 audio and a little more than #600 
video programmes as they are being used as supplementary materials to the main medium, i.e. 
print. As mentioned, they do not form part of the learning package despatched to the students. 
However, they are available to the students at the Study Centres - using them or not using 
them is left to the individual choice of the students, though every effort is made to impress on 
them the benefits of using them. Further, during face-to-face counselling sessions, they are 
played and discussed. But the general opinion of the students and counsellors is that they need 
to be made more effective. Though right now no authenticity can be established for this claim, 
it should be incumbent on the university to evaluate the effectiveness of these programmes. 
Though some sporadic studies, at individual levels, are available as regards the extent of use 
of these programmes, the result of which is not very encouraging, no serious attempts have 
been made to study the effectiveness of these programmes. Studying the learning 
effectiveness of these programmes or lack of it would help the university to rationalise the 
extent of their use. 

The same is applicable to the provision of teleconferencing. Though this facility is used 
periodically, the evaluation of its use for the students, etc., has yet to be undertaken. The 
sooner this study is taken up as a policy decision the better, because the cost involved is quite 
prohibitive if it is not put into proper use. The provision/facility may be good pedagogically, 
but the implementation of which is what the university should concentrate on. 



6.4 Evaluation of student support services 

Evaluation of student support services includes the following: 

i) tutor evaluation: Tutors/ Academic-counsellors are essentially engaged in the activities 
of informing, advising and counselling students both in the face-to-face contexts and at a 
distance by post; meeting students in face-to-face situations for a number of sessions during 
an academic year; assessing assignment-responses and helping distance learners to learn how 
they should learn on their own. These activities are the bases of evaluating the tutors. This 
evaluation, thus, consists in evaluating 

• the nature of communication between them and the students - through letters and/or in 
face-to-face situations, 

• the quality of their comments on assignment-responses, 

• the reliability and validity of the assessment of the responses, 

• the turn-around rate of assessed assignment-responses. 

ii) evaluation of face-to-face sessions: As far as face-to-face sessions are concerned, 
IGNOU evaluates the academic-counsellors’ punctuality, regularity, nature of rapport with 
students, quality of the conduct of sessions, students’ satisfaction, academic-counsellors’ 
motivation/enthusiasm, etc. 

Hi) evaluation of continuous assessment: This entails evaluating the validity and reliability 
of continuous assessment. This in effect means evaluation of assignments for their validity, 
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their reliability and for their correlation with the questions set in the end-of-the-term 
examination. 

iv) evaluation of support system: Evaluation of the support system itself consists in 
evaluating the quality and quantity of support available to students in relation to various 
courses at various study centres. Such evaluation will have implications for recruitment of 
academic-counsellors, providing support in terms of equipment and other support to counter 
issues relating to local and socio-geographical constraints, etc. The area of recruitment of 
academic-counsellors and their preparation for IGNOU purposes is elaborately dealt with 
here. 

Since 1987, the Indira Gandhi National Open University (IGNOU) has been conducting 
counsellor orientation/training programmes at its Regional Centres across the country. ( In the 
IGNOU context, counsellors are part-timers who facilitate face-to-face interaction). By and 
large, the number of these programmes organised in a given year largely corresponds to the 
number of courses launched in the immediate past and/or being considered. For every course 
at least two counsellors are identified, though this is not a norm. Full-time faculty of the 
university - both at the Headquarters and at the Regional Centres - conduct these tr aini ng 
programmes, the duration of which is 2 or 3 days. Normally, training/orientation programmes 
are scheduled on different months or different dates in a month in order to facilitate the 
movement of the core-faculty from one Regional Centre to the other. Approximately, 25 
trainee-counsellors, representing most of the Study Centres that come under the Regional 
Centres concerned participate in each training programme. As of May 1996, the total number 
of part-time counsellors, most of whom are tertiary level teachers in the conventional 
educational set up, are 14,542, of which nearly 50% have undergone the training. (The student 
enrolment in the academic year 1995-1996 is 1,30,228 and the cumulative number of students 
on rolls is about 400,000 - about 6% of the total number of students in higher education in 
India. During the 9th Five Year Plan, the university along with the other distance education 
systems in the country is mandated to cater to nearly 50% of higher education population. 
Plans to carry out this mammoth task is under way in right earnest). 



Selection of academic-counsellors 

Appointment of personnel in various fields, by and large, is made depending mostly on the 
educational qualifications without taking into consideration skills, aptitudes, attitudes, etc. 
Usually, the process of the appointment of counsellors at IGNOU reflects this contemporary 
social reality. This is not to deny the significance of content knowledge. But content expertise 
alone cannot be considered a determining factor. Typically, what happens at IGNOU is that a 
faculty which launches a programme informs the student service unit of its requirement in 
terms of counsellor-inputs. This unit, in turn, sends out communications to all its branches 
across the country for identifying counsellors for the programme. Though it is customary to 
specify educational qualifications for the appointment, there might be occasions when these 
are not prescribed leaving the option to the liaisoning agencies, i.e. Regional Centres. This 
initial responsibility of identifying potential counsellors invariably rests with the co-ordinators 
at the Study Centres. 
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A diagrammatic representation of the process is given below: 
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Fig. 4: Customary procedure to select IGNOU counsellors 
Issues to be looked at in this context are 

• the level of communication between the faculty concerned and the student service unit: a) 
does the faculty prescribe any other criteria besides content expertise, represented by 
academic qualifications, for appointment of counsellors? b) what role does the student 
service unit play, besides being a conduit point between its service branches and the 
faculty? 

• the role being played by the liaisioning agencies/service branches: a) do they involve 
themselves in identifying potential counsellors? b) do they play any role in the final 
selection processes, besides forwarding the bio data sheets? 

• the criterion used for selection of counsellors: a) can content expertise alone be adequate 
for a teacher to perform the functions required of a counsellor? b) if it is considered 
adequate, what is the rationale behind this supposition? c) if it is not so considered, what 
might be the other requirements and how can they be imparted? 

The operational procedure illustrated here does not appear to need any change and 
therefore can be retained. However, the nature of work of each of the operational units should 
be defined properly. In a pyramidical power structure of the kind IGNOU inherits, 
maintaining free flow of information, which is considered a crucial factor for total quality 
management across Schools and Divisions is difficult. Consequent on which, the faculty and 
service units, represented by Schools and Divisions respectively, seem to operate 
independently of the other, which gets manifested in the wide gap between the industrial and 
the servicing aspects of the institution under discussion. The situation does sound distressing. 
Increase in faculty-interest in student support servicing is the only way through which the 
servicing component can be strengthened.. 

The change being envisioned should thus originate from the Schools themselves which 
plan to launch programmes. A natural tendency for the Schools is to feel complacent with the 
production of course materials, which represents the manufacturing industrial process with its 
systems approach; division of labour; mass production, etc. Production of course materials 
should not, however, be considered the end of the distance teaching activity; it is just a part of 
it and in fact a beginning. In other words, the faculty should also look into the service 
industrial process of the institution. 

One way of getting the full-time faculty involved in the servicing sector is to make them 
participate in the selection and appointment of counsellors, who generally are part-timers. The 
institution should consider the following steps for this purpose: 
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Step One The faculty members of a particular School should ‘reflect’ individually on the 
processes that have gone into the production of learning materials such as why a particular 
topic/theme has been highlighted, for example, in a particular course-unit; what a theme 
intends to convey to the learners; why a particular pattern is followed in the arrangement of 
topics/themes; the areas which might pose some conceptual problems to the students, etc. 

Step Two Those involved in the production (or even other faculty members, depending on the 
need) collectively should discuss individual ‘reflective sheets’ and arrive at a consensus. (For 
example, why something has been done the way it is.) 

Step Three Based on the consensus arrived at earlier, i.e. steps 1 and 2, in step 3, a similar 
process should be repeated to decide on what one expects from an academic counsellor. And 
on this basis, guidelines should be prepared as regards the prerequisites of a counsellor. 
(Naturally, content expertise that currently is the determining factor for counsellor 
appointment will become just one of the many factors.) 

Step Four In this step, the list of guidelines prepared for the selection of counsellors, 
following the first three steps suggested above, is to be sent to the student service division for 
onward transmission to its liaisoning agencies, the Regional Centres. 

Importantly, the whole process being suggested here warrants that the interests of the 
clients, i.e. students, should be kept at the centre of the activity. 

The number of counselling sessions and the nature of content areas should also be 
discussed and rationalised. For this purpose, the student service division should liaise with the 
School(s) concerned. This is to see whether or not the demands of the School(s) can be met. 
The implication is that both the School concerned and the service division should be able to 
appreciate local problems, if any, at the Regional Centres, such as availability of expertise, 
etc. Ideally, interaction with the Regional Centres should be carried out when the Schools and 
the service division are in the process of discussing the issue. This would help sort out the 
constraints and act accordingly. This further suggests that right from the beginning, all 
participatory units will be involved in planning the counselling sessions. 

The whole process thus helps identify the right type of academic-counsellors. The role of 
the Regional Centres in this context involves the following: 

1 . Having received the list of requirements of the School concerned through the service 
division, the Regional Centres should communicate the same to the field offices, Study 
Centres and ask for a list of potential candidates. (As in the present practice, the initial 
identification will continue to be the responsibility of the co-ordinators of these Study 
Centres). 

2. On an appointed day, the candidates should be met in their respective Study Centres by 
the full-time faculty members who are at the Regional Centres. This initial meeting is for 
the purposes of (a) ice-breaking and (b) getting to know the attributes and attitudes of the 
candidates towards, for example, the open/distance education system, students in general, 
counselling, etc. (It is therefore necessary for the Regional Centres to keep ready 
interview-schedules and the like). 

3. With the introductory note on each of the candidates and, if possible, with a taped 
conversation, academic staff at the Regional Centres should meet and discuss each of the 
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candidates. This is mainly to facilitate objectivity in the selection. (Even marks can be 
awarded to each of the items that are being looked for from the candidates). 

4. After receiving the approval from the School and the service division. Regional Centres 
should issue appointment orders for a period of one year which can be renewed, 
depending on the performance of the counsellors. (It should be mentioned here that even 
at present, appointments are made only for a year initially and are renewed subsequently. 
The suggestion, therefore, may not be novel. What is being driven home, however, is that 
the Regional Centres should have some criteria to monitor the performance of a counsellor 
and accordingly take a decision on the question of renewal. By implication, right now 
such criteria-statements as mentioned are not available). The involvement of the Regional 
Centres in the process of counsellor-selection and their interaction with the counsellors 
will be more than what it is at present. 

5. On receipt of the acceptance of the offer from the candidates. Regional Centres should 
send them separate booklets on such areas as distance education; a profile of the 
institution; assignment evaluation; use of media; counselling; the role of a counsellor; 
clientele type, etc. They can be sent along with ‘reflective sheets’ in instalments. It is 
important that the counsellors complete the task of going through the materials and 
sending back their ‘reflective’ questions within a particular period of time. In the main, 
this ‘reflective’ exercise will help the trainees (and trainers as well) stand back from what 
they have been doing and think about what it means for their own learning and what it 
entails for their work as ‘teachers’ of others. Obviously, most would find this a difficult 
task to complete. (There is a need to be cautious in assuming that the written comments 
accurately represent the extent of the reflective thinking which the task may have 
prompted). 

6. The reflective questions should be discussed at the Regional Centres. This helps (a) to 
prepare a profile of counsellors (b) to identify the materials essential for training and (c) to 
decide on the content and focus of training. (By implication, the present system of a 
generalistic type of training would be inadequate. Depending on the subject and 
counsellors, the nature of training varies). 

7. Counsellors should be asked to attend at least three actual counselling sessions before they 
are invited for training. On attending the sessions, they record their observations in the 
‘journals’ they are asked to keep. These observations should also be sent to the Regional 
Centres. Moreover, those counsellors who actually conduct the sessions would also be 
sending their ‘journals’. These should be discussed during training. Some of the ‘senior’ 
counsellors locally available should also be involved. 

The whole process clearly indicates that even before the face-to-face training commences, 
activities of this kind prepare the trainee-counsellors for the task they would be assigned. This 
being what it is, the training scenario will change considerably. One instance is that more time 
will have to be spent on discussion/sharing of experiences. Obviously, however, to perform all 
that has been suggested, the trainers themselves should be resourceful. 

The ad hoc'\sm that seems to be prevailing currently in the appointment of counsellors, 
training per se, trainers and counselling will, therefore, undergo a qualitative change and a 
sense of professionalism can thus be infused into the whole activity. In the case of students, 
along with the customary offer-of-admission letters, a questionnaire in order to collect their 
expectations from counselling sessions should be sent. On the day of ‘induction’ they should 
be met with an analysis of these responses. This would immensely help all those involved in 
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counselling get useful insights into what needs to be provided/expected from face-to-face 
interaction. 

The implication is that before the actual counselling session begins 

• the counsellors are identified; 

• initial ‘cutting of ice’ would have been over by the time face-to-face training commences 
and that changes the orientation of the training; and 

• students would have been given an orientation so as to prepare them study with ease in the 
open distance education contexts. 

The process, however, does not stop here. The counsellors would interact with the 
Regional Centres continuously through their ‘journals’ and would in turn receive feedback on 
them. Every month, at least one counselling session conducted by a particular counsellor 
should undergo this journal study. Besides, every three months, all the counsellors should be 
met at the field level. The meetings can be recorded and discussed at the Regional Centres. In 
essence, it is a continuous process. This would help 

• identify the problems of the counsellors/students, etc. 

• identify a common pattern, if any, of the queries, problems, etc. 

• identify areas for the ensuing training 

• provide feedback to the institution on various aspects 

• establish a rapport with the counsellors/students and infuse a sense of belonging 

• the faculty members at the Regional Centres take up short-term research projects for local 
purposes. (The findings can be circulated to other Regional Centres to facilitate exchange 
of ideas.) 

• the full-time faculty to become professional trainers. 

Obviously, the list is not intended to be exhaustive. 



Orienting academic-counsellors 

The components of a typical two-say, approximately five hours a day, counsellor-training 
programme in the present context are: 

• the concept of open/distance education: the implication/relevance of these concepts in the 
higher educational scenario of India; the feature of open/distance education, etc. 

• the role of assignments and the importance of tutor-comments on the assignment- 
responses: assignments as a teaching tool (and not merely a tool for evaluation); how to 
comment on assignment responses, etc. 

• the notion of academic counselling: the difference between what is called ‘counselling’ 
and teaching; the differing role domains of teachers, etc. 

• specific content subject areas: how materials have been prepared; the role of self- 
instructional materials, etc. 

Though the orientation of counsellors is a regular feature of the student support activities 
of the university, embarrasingly, little efforts have so far been made to evaluate the 
effectiveness of these programmes for which the time-spent and money-spent are quite 
stupendous. The training programmes currently on offer demonstrate a ‘buckshot’ approach to 
training which often leads to a lot of irrelevant material in the programme fostering 
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dissatisfaction and instilling boredom among participants. Most of the counsellor training 
programmes conducted seemed to have been filled with information that different Schools and 
Divisions of the university think counsellors should know. IGNOU has yet to tailor this 
information to suit the needs of the participants. They are generally left to sift through what is 
presented to them and pick out what is relevant. Failure to profile the trainee-counsellors and 
accommodate the needs of both the participants and the organisation render the training 
programmes ineffective. Of late, however, it has been widely felt in the university that 
evaluating training in isolation will not help serve the purpose of providing effective training. 
Every aspect of training and related factors should, as a rule, be paid attention to. This 
includes evaluating the procedure involved in counsellor-selection because selecting the right 
people for the right job is important. 

In general, to evaluate student support services, IGNOU seeks answers to a few questions 
of the kind given here: 

• do these services cater to the information needs promptly, adequately and convincingly? 

• do these services provide for advice at pre-course, on-course and post-course stages 
adequately and effectively? 

• do these services provide for academic-counselling adequately and conveniently? This 
includes the effectiveness of tutors with regard to their work on assignment-responses and 
face-to-face situations. 

• are these services easily accessible? 

• are the staff involved in these services attitudinally tuned to the kind of work assigned to 
them? 

• how do these students rate the support services? 

• do the support services make a pedagogical ly rational use of various media in operation? 

• do the support services help the learners get a good pass in the examination? 

The criteria that have been listed above bring in a host of variables - human beings in the 
roles of co-ordinators, academic-counsellors, etc; geographical situations such as towns, 
remote areas, rural areas, etc., and pedagogic factors such as classroom techniques, 
counselling, advising, etc. Obviously, evaluation of support services is a difficult area and , 
therefore, IGNOU has to pay more attention to this area than it does at present. 

7 Evaluation for accreditation 

As mentioned, in India, there are 7 state- /province-level open universities and a little more 
than 50 distance education/correspondence education units attached to the conventional 
universities, besides IGNOU, at the national level. A few of the total 26 states/provinces are 
keen in establishing open universities. (In fact the Central Advisory Board on Education 
(CABE), the highest educational policy-making body in India has recommended that in the 
9th Five Year Plan (1997-2001) every state/province should have an open university). 

The tremendous growth of distance education in India thus is imminent. However, at 
present, various institutions use different types of instructional systems. Moreover, similar 
kinds of programmes are offered at every institution. By implication, duplicatory efforts are to 
be removed. For this purpose, various activities need to be undertaken, some of which are 
listed here: 
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• the materials of various institutions should be evaluated so as to assign particular tasks 
(academic disciplines) to particular institutions, 

• facilitate transfer of credits among institutions making possible the movement of students 
from one institution to the other, 

• print materials should be transformed into self-instructional materials, 

• electronic media, face-to-face sessions and assignment-responses to facilitate two-way 
communication should be incorporated. 

This is essentially the task of the DEC. Right now, despite its existence for about four 
years, this task has not been in full operation, mainly because of some structural and 
procedural constraints. However, a few attempts have been initiated in sensitising the 
authorities and faculties of the conventional universities in the area of distance education. For 
example, STRIDE organises from time to time training-workshops for the academics at the 
state- /province-level open universities and distance education units of the conventional 
universities. The focal point of these training-workshops is on transformation of existing 
materials into self-instructional materials, as this is the immediate concern. Besides, 
workshops for training the trainers are also conducted in order that each distance education 
unit will have a few resource persons to train others and supervise the task. Simultaneously, 
DEC is also in the process of pooling together of materials available at various institutions for 
purposes of resource-sharing. Recently, a manual has been prepared for providing guidance as 
regards instructional systems/strategies for use at distance education institutions. 

Criteria for evaluation of the instructional systems in general and self-instructional 
materials in particular have to be evolved. One possibility is to benchmark the materials 
against the IGNOU materials. But, for this purpose, IGNOU materials have to be evaluated 
first in terms of effective student learning. The status being what it is, it would easily take a 
few more years to actually evaluate the materials for purposes of national accreditation. 

$ Evaluation for financial backup 

As a funding agency for distance education systems in the country (at par with the UGC for 
conventional universities), IGNOU through DEC is expected to make funds available to 
various institutions for projects of various types ranging from curriculum development, 
launching of new academic programmes, staff development - both academic and non- 
academic and so on. This process has already started and right now some state-level open 
universities are the beneficiaries. The benefits will be extended to the distance education units 
attached to conventional universities in the years to come. However, the funding has been 
provided on ad hoc basis. But this is to be changed and more systematic procedures and 
norms for funding have to be evolved and streamlined, lest the purpose for which it is 
intended may not be served. Work towards this goal is being undertaken. 

9 Evaluation to assess social impact 

Nothing tangible has been done in this regard. However, there is a growing concern for this to 
happen both within the university and outside including the government. Soon this activity 
will assume the centre stage in the IGNOU scheme of things. 
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For purposes of discussion, by social impact is meant seeking answers for such questions 
as (Koul,1991) 

• are the students satisfied with what they have achieved as students of the university? 

• can the students sell their services on competitive terms in the employment market? 

• are the students rated favourably (if not better) against the students of comparable level 
from a conventional university? 

• do the students succeed in contributing to their social and economic upliftment in 
particular, and that of the society in general? 

• do the students find easy access to further education in the conventional system, if they 
decide to turn to that system? 

• do the students attract social respect and recognition? 

One can add a few more to this list. But what is presented here is a representative sample 
of the issues that IGNOU should address itself to for the purpose of impact evaluation. 

10 Summary 

Starting with a section on ‘background’ which presents the emergence and mandates of the 
Indira Gandhi National Open University, India, this paper discusses the evaluation practices in 
use at the university. For this purpose, the paper is structurally divided into seven thematic 
sections each presenting one type of evaluation practice with one section on the list of 
evaluation types as used at and assigned to IGNOU. In this paper, we have, thus, discussed 
evaluation for curriculum development/design, for student entry to various academic 
programmes, evaluation of student performance, programme evaluation, evaluation for 
purposes of national accreditation and of granting funds and social impact evaluation in that 
very order. The paper observes that IGNOU has yet to streamline much of the processes of 
evaluation. 
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I. Introduction 

This document examines the evaluation process of distance education at U.N.A. ( Universidad 
Nacional Abierta de Venezuela - National Open University of Venezuela). Evaluation here 
refers to the two aspects involved; that is instruction and administration as conducted at 
U.N.A., which is a higher education institution that offers an efficient, highly productive and 
low-cost way of covering the growing demand for higher education. 



II. Background 

In response to the above mentioned growing demand, in Venezuela a record number of 
students have gained access to higher education through the distance education program 
developed and implemented by distance educators at U.N.A. 

The U.N.A. is composed of various sub-systems aimed at achieving the institution's objectives 
as well as those of the student population. Therefore, it possesses flexible characteristics and a 
capacity for self regulation. This ensures a progressive up-grading of its activities and 
products. As we shall see later this capacity influences the evaluation process. 

U.N.A. therefore is oriented by institutional principles and objectives. These are: 

II. 1 . Institutional Principles 

• Democratization 

• Educational Innovation 

• Individualized Instruction 

• Optimization of the Investment 

• National Character 
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II.2. Objectives 

• Institutional Objectives 

• Functional Objectives 

• Operative Objectives 



The principles of U.N.A. may be defined as follows: 

• Democratization: U.N.A. offers opportunities of higher education to students of diverse 
social strata, in particular, to those who are employed, or have not had the opportunity to 
pursue higher education in the traditional system. 

• Educational Innovation: U.N.A. possesses structures capable of developing and 
incorporating innovations tending to optimize the learning process. 

• Individualized Instruction : The learning system at U.N.A. is based upon individualized 
study, which is relevant to the conditions, needs and aspirations of the students. 

• Optimization of the Investment: U.N.A. must contribute to a significant reduction of the 
costs per student as well as the cost per graduate. 

• National Character: U.N.A. extends its higher education services to the entire national 
territory according to the demand for education. 



With reference to these principles, the objectives of U.N.A. have been classified in three 
groups: 

Institutional Objectives: 

• To train human resources for priority areas of socio-economic development 

• To train professionals who will act upon the social system as agents for the qualitative 
change demanded by the nation. 

Functional Objectives: 

• To develop, implement, and administrate new strategies in the learning process. 

• To carry out research, to evaluate experiences in the new fields of open and distance 
education. 

Operational Objectives: 

• To optimize the efficiency of educational investment. 

• To cooperate with regard to enrollment in formal higher education so that the latter may 
increase its efficiency and effectivity. 

© r\ r*. 
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Finally and for the purpose of this paper we define the following terms: 

Distance Education: Implies the utilization of media systems, either individual or collective, 
in order to offer effective instruction, especially where the direct contact between students and 
educators does not exist. 

Open Education: Constitutes the removal of academic restrictions and privileges: the 
elimination or reduction of the barriers between areas of knowledge, careers, institutions, the 
increasing and enriching of useful activities and experiences to complement the academic 
educational purposes, and above all, proposes substantial changes in the traditional 
relationship between the students and educators. 



III. The Problem 

Evaluation is defined as the act or result of examining and judging the value, quality, 
significance, amount, degree or condition of something; and the purpose for evaluating such 
an institution as U.N.A. is to estimate quality, relevance, performance, etc., with a view to 
correct, improve or change as well as to keep track of the academic evolution of the 
university, to build a reliable data base and devise a set of criteria and adequate mechanisms 
for institutional support. The set of data systematically gathered shall provide a basis for range 
of activities carried out by the different administrative and academic units of the University. 

Therefore evaluation should be considered as a process that implies a comparison of the 
object under evaluation to another similar object used as a standard of comparison whose 
qualities are well-known to evaluators. Its operational aspects present difficulties, because the 
evaluation of a distance education system, being dynamic, requires permanent monitoring and 
restructuring. This, in turn, is due to the fact that the very target of the evaluation is 
undergoing constant changes. 

The above mentioned standards of comparison could be either quantitative or qualitative: in 
both cases they are man-made and, therefore, as evaluation criteria they cannot have universal 
value; on the other hand these standards define an ideal state, an acceptable or anticipated 
behaviour, an intended result or goal, etc., which, in turn, implies the need for collecting 
relevant information on both the exact state of the object for evaluation and the criterion (ideal 
state) to be used for comparison. 

The difficulties involved in both the selection of all necessary information and the comparison 
itself are something worthy of special attention. At U.N.A. it was concluded that the 
evaluation system has to have a proper identity and sufficient independence so that it may 
both perceive and demand compliance with the principles and objectives of the University. 

In short, it is a system that on the one hand provides an overall view of the university as a 
whole, paying attention to the present situation, to the evolution and prospects that apply to 
academic areas and to results so that they can be used for purposes of institutional support and 
related administrative activities. The evaluators should, then, be members of the academic and 
administrative staff who have deep understanding and knowledge of the institution - and 
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therefore it was decided that the evaluation would be carried out only by members of the 
academic and administrative areas of the U.N.A. 

Therefore we may conclude that the institutional evaluation of U.N.A. possesses the following 
general characteristics: 

It is an institutional self-study performed by staff; the purpose of such study is to evaluate 
learning, attitudes, attrition, cost-effectiveness, staff effectiveness and course design; the key 
elements are: committee work, standards set by staff discussion; professionalism. 

The risks involved are: alienating some members of staff and ignoring the values of outsiders, 
but it has as a positive result the increase of staff awareness and the sense of responsibility. 

The above mentioned committees were formed on a voluntary basis and there were no 
restrictions imposed on the staff; thus they were able to choose the area where they would like 
to work. Two major areas were defined - instruction and administration. Each of these areas 
shall be analyzed in detail by a committee named the Evaluation Committee for Instruction, 
and Evaluation Committee for Administration. The aim of the committees is implied in their 
names and they will be carrying out a number of activities in connection with the project that 
is hoped to be be under way in 1997. 



The Instruction Area includes the following sub-categories: design and validation, teaching- 
learning process and student evaluation process. 

Design and Validation Process. The Curriculum Design determines the profiles, pedagogical 
objectives, and tasks. These curricular specifications are transformed by the Instructional 
Design into programs suited for learning and "instructional sets" for the student (written 
media, audio- visuals, etc). These sets are validated on a representative sample of the 
population before being applied to the courses. 

Teaching-Learning Process. The relation between the student, the learning materials and the 
special teaching personnel, initiates the phase of interaction of teaching-learning. The student 
- subject and focal point of the program - studies and carries out the actions established for the 
achievement of the teaching objectives (readings, consultations, experiments, use of audio- 
visual media, study groups, etc.). 

Student Evaluation Process. The characteristics of the system urge the student towards 
constant self-evaluation, so that he has ample and varied opportunities to measure his progress 
and limitations, and of overcoming the latter with help of (‘face-to-face’) tutors or advisors, 
before submitting himself to the intermediate or final examinations. Permanent self-evaluation 
is complemented by the rigorous and controlled evaluations carried out by the University. 
From this phase, information produced by the summative evaluation is transmitted to the sub- 
system of student evaluation. 

The above mentioned self-evaluation will give the student the opportunity to perceive his 
basic knowledge, the most difficult objectives to attain, and the aspects that will be necessary 
to put emphasis upon his learning process. 
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The characteristics of the student-evaluation process as controlled evaluation carried out by 

the Institution are: National, simultaneous, objective, dynamic, flexible, informative, 

publicized, economical and efficient. In addition, the evaluation process implies students’ 

obligation to attend. 

• National since it is administered nation-wide and the student may take the test at any Local 
Center. 

« 

• Simultaneous since every step of the process is unique and applied in the same way at all 
Local Centers at the same day and time. 

• Objective since it is determined by and emphasizing the features of the subject matter, 
rather than the thoughts, feelings, etc of the professor, content expert, student, etc. 

• Dynamic since it is a vigorous system tending toward change at every step of the process 
development (see Flexible). 

• Implying obligation to attend since all tests require students’ attendance. The tests are 
applied by the University and contain all the elements considered necessary to judge the 
attainment of the learning objectives. 

• Flexible since it is adjustable to change and capable of modification (see Dynamic). 

• Informative since at the beginning of every academic term the student is well informed of 
the details of the tests, that is, contents, day and time of day of administration, etc. (see 
Publicized). 

• Publicized since it is known by or open to the knowledge of all students accross the 
country by the use of billboards at the Local Centers. 

• Economical since it is conducted in an economical manner, that is, not wasting money and 
time (see Efficient). 

• Efficient since it is developed in such a way that produces the desired effect or result with a 
minimum of effort, expense or waste (see Economical). 



The Administration Area comprises all ordinary and routine activities of administration, 
budget control, personnel and services. This area is divided into three units: General Services, 
Personnel and Budget Control. 

General Services. Process of internal administration with respect to material needs, 
commercial functions, maintenance cooperations, filing services, reproduction of material and 
internal communication. 



Personnel. Carries out the administrative processes of the different development programs 
and control of personnel. Comprises the functions of recruitment and selection, registration 
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and control, classification and remuneration. 

Budget Control. Register of expenditures, comprises the general accounting process and the 
register of debits against the annual budget. 
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Introduction 

British Columbia is the size of western Europe and has a population of just over three million 
people, two-thirds of whom live in the south-western comer of the province near the border 
with the USA. The other one million inhabit a large number of very small towns and villages 
set in mountain valleys, along rivers and lakes and on coastal islands. It is among this one 
million people that one finds those depending upon distance education. 

British Columbia has four universities, three of which, also located in the south-western 
comer of the province, were constructed as traditional institutions with a strong sense of 
place. During the 1970’s and 1980’s, as the job market began to change away from complete 
dependence on primary industry, and politicians and bureaucrats began to identify the 
growing needs of the one million British Columbians living in rural areas or small towns, 
approaches were made to these three universities to provide programs at a distance. The initial 
response was varied. The University of British Columbia showed little interest. Simon Fraser 
University showed a lot of interest. The University of Victoria was somewhere between these 
two positions. (A more detailled description of The University of Victoria can be found in the 
Appendix.) 



A Perspective on Philosophy 

The value of a paper describing how the traditional University of Victoria has developed in 
twenty years into a substantial distance education provider, probably lies in confirming the 
common experience of many similar institutions and perhaps providing some guidelines for 
institutions yet to begin this almost inevitable journey into an uncomfortable world. 

Why „uncomfortable“? Because the traditional academic, for whom teaching is a matter 
of facilitating a student’s conversion of information into knowledge - a subtle task - the 
delivery of what is, in most cases, simply packaged information, is neither challenging nor 
particularly valuable. There exist in fact many academics who have yet to be convinced that 
the delivery of information to an isolated student is a worthwhile experience, and the literature 
is of little assistance in changing this perspective. 

Regarding the evaluation of distance education, it may be observed, therefore, that at The 
University of Victoria the traditional concerns about the learning experience of the individual 
student have been carried into the distance education process and this individual remains 
today the primary focus of distance instruction and evaluation. 

A summary of The University of Victoria’s experiences with distance education may help 
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to put evaluation into perspective. In the 1960’s the university established a Division of 
Continuing Education, whose primary purpose was to respond to educational needs identified 
by campus-based academics, many of whom had little or nothing to say beyond suggestions 
for the establishment of off-campus versions of existing courses. This all began to change 
with the arrival on-campus of Dr. Glen Farrell, a professional adult educator, unlike most 
continuing education specialists whose academic credentials lay elsewhere. Farrell is currently 
the President of British Columbia’s Open learning Agency, widely regarded as one of the 
most effective distance education institutions in the world. 

Farrell’s arrival at The University of Victoria coincided with the first attempts in Canada 
to launch communications satellites, and with the advent of the ANIK series of satellites, The 
University of Victoria found itself part of British Columbia’s Satellite Tele-Education Project 
(STEP), an ambitious collaborative project which tested the possibilities of satellite-based 
delivery of educational programs. 

It was at this point that evaluation of instructional design and delivery became important. 
What was evaluated? Technological systems and instructional design models. Not students. 
The evaluation process focused on the experiences and opinions of those who had designed, 
produced and delivered the instructional experiences, and those who had managed the whole 
program. 

The result of this fascinating experience was that the consortium of British Columbian 
institutions of which The University of Victoria was a member, was awarded transponder 
space on a satellite and an era of high-tech delivery of educational services began and has 
expanded ever since. A second result, specifically at The University of Victoria, was the 
inclusion of television and audio-conferencing in the list of useable distance delivery 
technologies. And a third result was a change in the status of the Division of Continuing 
Studies to a more autonomous, pro-active unit which began to initiate distance-based 
programs and to promote distance education as an equal delivery system, rather than the 
servant of campus-based academics. 

By the late 1980’s interactivity and communication between distant student and campus- 
based instructor began to dominate the instructional design of distance education courses and 
programs, and computers were being explored as delivery media which might enable this to 
occur. And through this decade The University of Victoria incorporated a substantial array of 
technology-based distance delivery options, including interactive video conferencing, and 
extensive networking. 



Methods and Procedures 

Concerns about how best to evaluate distance education at The University of Victoria have 
changed and grown as the institution has become committed to distance education. The 
concerns about design and technology remain strong, although Schramm’s observation in the 
1970’s that most technologies work in some circumstances, has been accepted and there are 
few attempts to evaluate instructional machinery. Course designs are now evaluated from their 
first beginnings. A fairly common method of doing this is based upon the assumption that an 
initial piloting of the design with one expert can usually identify about 80% of potential 
problems. This is a simple but effective method which originated with traditional classroom 
teaching. A second pilot with a small group of students is then conducted and a student 
evaluation of the design is obtained, from which the proposed course is fine-tuned. 
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However, as the years have passed and several university departments have become 
involved in distance education, including The School of Nursing, The School of Social Work, 
and The Faculty of Education, two characteristics of The University of Victoria’s approach to 
evaluation have become apparent: the fact that virtually all distance education at the university 
is generated by the professional schools for working professionals whose needs are 
continually changing; and that the focus of evaluation is on individual student experience. 

How has this occurred? First the concentration on professional schools. In a study 
completed in 1985, with colleagues at The University of Calgary, this author learned a few 
things about who needs distance education and what types of courses lend themselves best to 
this instructional mode. Much of what seemed to be revelations then, appears obvious now. 
Courses requiring the intense interaction between one who knows and one who learns, almost 
like apprenticeship, do not deliver well via many currently available technologies. While 
some new technologies show promise, even the most interactive fail because of poor detail, 
uncertain colour and limited screen space. 

Secondly, courses requiring high degrees of interaction with information do not deliver 
well via current technologies. 

Thirdly, the group of workers within British Columbia that most consistently and 
continuously needs upgrading and in-service training are the members of professions: nursing, 
social work and teaching, law and medicine. 

The geography of British Columbia and the isolation of distance learners has therefore, 
influenced the shape of evaluation towards a focus on the experiences of the individual 
learner, possible in British Columbia because distance education course populations are 
usually small and often comprised of students separated by considerable distances (fifteen 
hundred kilometres is not unusual). So individual support becomes important, and even a few 
drop-outs can cause a course to be withdrawn. Evaluation of courses offered by the university 
under these circumstances usually takes one or more of the following forms: 

(i) a questionnaire to the student focusing on individual learning experiences; 

(ii) a questionnaire asking the student to consider some of the elements of the instructional 
design of the course; 

(iii) assessment of the student’s mastery of course content over a substantial period, i.e. as 
much as one year, to test for retention; 

(iv) an assessment of the ways in which the student has applied the course content in 
professional activities. 

While this process may provide evaluators with information about the student there is, 
and always has been, one weakness in this evaluative method: it assumes that the student 
knows how to learn at a distance. Since there is little evidence that this is so, information 
received may not provide evidence of the instructional design effectiveness since no-one can 
be certain of the extent to which the student was able to learn via this delivery method. 



The Current Situation 

Currently at The University of Victoria, evaluation of distance education programs remains 
dependent primarily on a formative process that focuses on the individual student. There is no 
written policy concerning this: it has evolved as a result of the university’s traditional, 
classroom-based instructional history. 
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The process has essentially two players: the instructional team, which consists of a 
designer, a media specialist, a communication specialist and a course-writer, who will design 
the course and conduct their own formative, internal monitoring; and an external evaluator 
who will watch the process, periodically consult and perhaps test, and conduct the summative 
evaluation. 

Here is a description of how this process works. 

1 . The process by which a course eventually finds its way into the homes of distance 
learners begins with informal discussions between the external evaluator and the 
instructional design team, to ascertain three things: 

(i) a collective understanding of the course’s overall goals and purpose; 

(ii) a specific understanding of the instructor’s view of the course; and 

(iii) identification of evaluation as the basis for continuous revision of the course design. 

2. From this point, the course design proceeds until a rough draft is ready to be presented to 
one expert for initial revision. 

3. Once it is ready, the course is pilot-tested with a group of students. At the mid-term point, 
the external evaluator and the instructional design team obtain information from the 
students about three things: 

(i) a general reflection on the course thus far; 

(ii) a more focused discussion on the students’ views of the specific design of the course; 
and 

(iii) a third discussion or questionnaire concerning the effectiveness of the mode of 
interaction between the instructor and the student. 

4. Student responses at the mid-term evaluation are used in two ways: 

(i) to improve the course; and 

(ii) to generate further questions regarding the overall organization of the course, which 
are added to a template of evaluation questions constructed from previous course 
experiences. 

Evaluations always focus on the student’s experience and how to improve it. 

This model is not unique to The University of Victoria. Many will recognize its similarity 
to that used by the British Open University and by various Australian institutions, in fact 
much of it is published in The British Journal of Educational Technology (1979-84). 

5. At the conclusion of the course, the external evaluator will not only set a questionnaire for 
the students, but will also assess the effectiveness of the design from the point of view of 
the design team. Typically the issues investigated here are as follows: 

(i) the effectiveness of the organization of the material; 

(ii) the effectiveness and appropriateness of course content; 

(iii) the amount of time taken to assimilate the materials and prepare the course; 

(iv) the quality of the learning activities; 

(v) the effectiveness of the student support system. 
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Examples 

The following are examples of the types of questionnaires given to students by evaluators and 
the types of questions asked in evaluations. 

Example 1. A Mid-term Questionnaire: 

Respondents', course students. 

Time: mid-way through the first pilot version of the course. 

Information to be Obtained from the Questionnaire: 

1 . The value of the course package as an aid to learning. 

2. The effectiveness of the course in helping a student to think critically about the issues 
presented. 

3. 3. The effectiveness of the course organization. 

4. Ways in which the reading materials add to a student’s understanding of course topics. 

5. The usefulness of learning activities in helping a student to understand course materials. 

6. The effectiveness of electronic audio and visual materials. 

7. Whether or not any difficulties students experienced in obtaining materials outweigh their 
usefulness. 

8. The effectiveness of the course package in setting out the learning activities. 

9. Clarity of assignments. 

10. Adequacy of the amount of time given for the completion of each course unit. 

1 1 . The extent to which the course 

- makes a student think about course issues; 

- makes a student feel part of the class; 

- stimulates group discussion; and 

- makes a student feel less isolated. 

12. The usefulness of course web pages in helping a student to master the goals and 
objectives of the course units. 

13. Suggestions for other ways in which the course can make use of web-pages. 

14. The extent to which course assignments help the student to learn course material. 

1 5. Whether or not the instructor’s assessment plan is fair. 

16. Whether or not the instructor is helpful. 

17. The best and worst aspects of the distance education course. 

Some of the most useful information in these evaluations is that which is given to guide 
the student in answering questions, for example: 

(i) Describe in your own words any changes, additions, deletions, rewordings which would 
improve the course. 

(ii) Reply out of your own experience in taking the course, thinking how it was for you, what 
was valuable, what might make it more valuable. 

(iii) You have been in a distance education course and you should not confuse it with a direct, 
face-to-face course. For example, to say that the course did not provide you with personal 
contact with the instructor and other participants, would not be particularly helpful, since 
no electronically-mediated or distance education course is designed to do that. 
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Example 2: A formal questionnaire used extensively in distance education courses: 

This type of questionnaire is used extensively in distance education evaluation. It seeks to 
provide feedback from the students to the course designers. 

Students are asked to rank a set of items on the following five-point scale: 

5 = outstanding 
4 = very good 

3 = satisfactory , 

2 = parts were not satisfactory 
1 = not satisfied at all 
0 = not applicable 

The set of items deals with course design, organization and management: 

1 . Adminstrative, registration, fee payment procedure 

2. Bookstore services 

3. Effectiveness of the course text 

4. Effectiveness of course organization in supporting learning 

5. Effectiveness of text assignments in supporting learning 

6. Effectiveness of assignments employing other media in supporting learning. 

7. Effectiveness of instructional procedures in supporting learning: 

(i) concept mapping 

(ii) simulation 

8. Effectiveness of feedback procedures on assignments 

9. Effectiveness of communication processes: 

(i) teleconferencing 

(ii) fax 

(iii) email 

10. Academic influence of the course 

1 1 . Personal influence of the course 



Evaluation of Students Studying at a Distance 

The only guideline provided for the evaluation of distant students is that they be judged 
according to exactly the same criteria and standards as those employed on campus. 

Members of The University of Victoria engaged in the delivery of course material by 
distance education have used a variety of techniques to evaluate students. 

Formative evaluation sometimes takes the form of self-administered tests incorporated 
into course materials, whose sole purpose is to provide students with personal feedback. 
Typically such a test, called an Info-Check, comprises a dozen or so incomplete sentences 
whose completion provides a means of comprehension monitoring for the student. 

The Info-Check is supplemented by a secondary monitoring device called an Activity 
Option, whose primary purpose is to cognitively extend the student’s understanding by posing 
a series of questions requiring the student to relate course content to personal and professional 
experiences. 

Neither of these two processes is graded. 
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Mid-term essays or products, sent directly to an instructor are also used as a means of 
student evaluation. 

Summative evaluations are usually conducted through an examination or a final course 
project, often submitted electronically. Some members of the university are currently 
exploring electronic testing as an alternative option, a process by which an off-campus student 
accesses a course examination by computer, completes the examination at the computer, and 
with the pressing of a return key, sends his answers directly to an instructor. 



Conclusion 

For a university which began life with a clearly defined identity as a traditional institution to 
which students come for face-to face instruction, the University of Victoria has moved a long 
way towards the dual-mode, flexible-learning instructional modes commonly found in open- 
learning institutions. British Columbia is well-served by an excellent open learning agency 
and there is no need for The University of Victoria to focus on the delivery of distance 
education beyond its present, evolving and growing commitment. 

Throughout its growing involvement with distance education, evaluation has played a 
critical role in shaping the relationship between the university and its distant students. While 
influenced by the early models of the British Open University, The University of Victoria has 
preferred not to develop too formal policies on evaluation but rather to rely on a set of 
guidelines which experience tells, work effectively most of the time. 

The most significant characteristic of the university’s evaluation process is the emphasis 
on the individual student - a legacy from the days of traditional classroom teaching, but also a 
factor of being located in a large geographic area with a small population whose individual 
students are each significant. 
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Appendix: Distance Education at the University of Victoria 

The University of Victoria first opened its doors in 1963. Since 1975 it has been a pioneer 
in distance education in Canada, having explored and tested many of the delivery modes 
currently used in distance education throughout the country, particularly satellite-based 
delivery, teleconferencing, and multi-media delivery. 

Currently the University's Division of Continuing Studies works with academic faculties 
in the design and delivery of distance-based instruction in the following areas: 

• Bachelor of Arts in Child and Youth Care 

• Bachelor of Science in Nursing 

• Bachelor of Social Work 

• Certificate in Adult Continuing Education 

• Education Degree Completion 

• Diploma in Public Sector Management 

• Diploma Program in Cultural Conservation 

• Certificate Program in Computer-Based Information Systems 

There are approximately 12,000 adult students in British Columbia who study via 
distance education, which is about 16% of the all adults studying at the tertiary level. 
Approximately 20% of these students take courses offered directly by The University of 
Victoria. The University of Victoria's distance education programs offer degree completion, 
professional development and personal enrichment courses to these students who are, 
predominately, adults with professional, career and family responsibilities who cannot attend 
university during the traditional hours, or who live in remote areas of British Columbia and 
Canada. 

Profiles of these students indicate that most possess professional skills, substantial work 
experience and career mobility. Distance education courses offered by the University enable 
these students to work at their own pace, in their own location and relate course content 
directly to their daily occupations. The courses may be structured or open. Most are highly 
interactive and many are supported by computer-based tutorial links. 

The University's faculties of Business, Education and Law, and its Schools of Social 
Work, Child and Youth Care and Nursing are the primary users of distance education delivery 
systems. The academic content of each program offered is the responsibility of an academic 
department within the University. The management and delivery of courses is the 
responsibility of the Division of Continuing Studies. Instruction is given by both full-time and 
part-time instructors. 

In British Columbia many of the courses offered by education institutions are 
academically transferable. Most institutions are electronically linked. Thus programs offered 
by The University of Victoria are complemented by programs offered by The Open Learning 
Agency of British Columbia and other universities in the province and the country. The 
University is also a member of NACER, a Nafta-funded consortium of nine institutions in 
Canada, USA and Mexico one of whose purposes is the distance delivery of courses across 
national borders. Courses offered through this consortium may also be blended into a student's 
program. 
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Introduction 

This article aims at giving an overview of ideas, principles and practices concerning quality 
assessment and evaluation at NKI Distance Education. The main emphasis will be placed on 
system evaluation - in both micro and macro perspective. The article discusses concepts and 
practices related to quality assessment and evaluation. NKI practices concerning student 
evaluation are also accounted for as one aspect of internal evaluation. It has been the intention 
to describe and discuss NKI practices and experiences, rather than writing a scientific article. 
The emphasis has thus been put on concepts, methods and experiences, not on presenting 
specific results from research and evaluation projects. This article represents an update of the 
contribution to the report on evaluation practices and experiences published by the 
FemUniversitat (Schuemer 1991, Rekkedal 1991). The Norwegian laws and regulations 
concerning governmental accreditation and control of distance teaching institutions were 
drastically changed in 1993. Some important aspects of this change are also discussed. 

NKI holds the view that evaluation has to be an integral part and continuous activity in any 
educational undertaking. Without evaluation and assessment of students during the learning 
process, conditions for learning are far from ideal. In the same way evaluation of the 
organisation, the teaching/leaming process, the learning material and media for learning is 
necessary for institutional learning and increasing quality. 

20 years ago, as a graduate student at the University of Oslo, the author of this article was 
planning the research project for his degree in education. He was interested in adult education 
and in correspondence education in particular. Until then very little research had been carried 
out within this area, and the author’s intention was to make a large scale pilot survey on 
recruitment, study achievements and completions/drop-out in correspondence study in 
Norway. More or less by incidence he came in contact with NKI and asked if he would be 
allowed to carry out the research based on NKI files of correspondence students. He was 
welcomed and given completely free hands in collecting the amounts and types of information 
he wished. A year later his thesis was completed. 

The information presented did not at all give a glorious picture of the results of the 
institution. Although the main focus of the research was to find relations between background 
factors, personal factors and situational factors with different criteria of success in 
correspondence study, one could not oversee the fact that the completion rates presented were 
low. The results were, in fact, quite disappointing, and one could guess that many 
administrators would have taken the decision to try to keep such information confidential. 

The rest of the story, in our view, gives a picture of NKI's views on institutional evaluation 
at that time. Instead of being asked to keep a low profile on publication of results, the 
administration decided that the full report should be distributed to all individuals and 
organisations involved in or specifically interested in correspondence education in Norway. 
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An abbreviated report containing all factual information was published in Norwegian and 
English and distributed nationally and internationally. The author was present once when the 
director was asked how he dared to publish the results of the drop-out part of the study, and 
the answer was that he did not dare not to publish, as openness of institutional results is 
necessary for increasing quality. The immediate reaction at the institution was that all 
employed - academics, administrators and clerical staff - became involved in a project aiming 
at assessing the organisation and administrative, teaching and counselling routines and 
procedures to state long-term and short-term goals for changes to increase the quality of 
teaching material and student services. At the same time the author was engaged to plan and 
conduct a research and evaluation scheme with the same goal - quality control and quality 
development of the NKI distance education system. Since then, research and evaluation to 
assure and increase the quality of the teaching system have been focused on as an integral 
activity in the development and administration of all distance study programmes. 



NKI today 

NKI is the second largest non-governmental educational institution in Norway. It is organised 
as a non-profit foundation with governmental control and financial support from the state. 
Today, the organisation has four different departments offering services to the public: 

The Polytechnic College was established in 1 995 by a merger of the two NKI Colleges, the 
College of Engineering and the College of Computer Science. The Polytechnic College offers 
two and three years programmes in computer science and engineering. In contrast to what is 
happening in many academic institutions today, where one moves from traditional academic 
studies into distance education and new modes of teaching, the NKI College of Engineering 
was originally developed from an existing distance education programme. The second of NKI 
units is Nceringsakademiet which offers vocationally oriented studies at secondary level as 
part-time and full-time programmes in several towns and cities. All the studies emphasise 
practical applications of computers. NKI Publishing House develops materials for both NKI 
activities and for sale on the open market. 

NKI Distance Education is the second largest distance teaching institute in Norway 
recruiting 7.000 to 10.000 students a year, or approximately 35.000 course enrolments. The 
programmes offered cover technical/vocational training, management, computer science and 
business training on secondary and tertiary level. 

To support the activities NKI has established a Research and Development Unit 
responsible for research in education and evaluation. The unit has given priority to research 
and development activities within distance education and was integrated into NKI Distance 
Education in 1993. 

Some elements and dimensions in the concept of evaluation at NKI 

Evaluation is part of most human activities involving decisions and simply means assessing 
the quality of something. We evaluate when we buy a suit or even a hamburger, when we 
decide to go to the theatre instead of the cinema, and when we decide which play to see. After 
the action we assess again to see whether we are satisfied or would make another choice at 
another time. 
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Systems approach/educational technology 

At the general level NKI follows the ideas and principles of educational technology in 
developing and delivering education. In general models of education and teaching, evaluation 
always plays an important part. Evaluation is the starting point for the feedback loop in the 
process. At every stage of development the results are in some ways formally or informally, 
subjectively or objectively assessed and necessary action taken. Ideally, but not as often as we 
would wish, a total education programme is offered in a pilot version for formal evaluation 
and revision before the programme is offered on a larger scale. The usual procedure in these 
cases is to market a new programme in a pilot vision which is tried out on a restricted group 
of students before the final version is put on the market. Later, as long as the programme is 
offered, successive evaluations and revisions are carried out and the programme is 
incrementally improved and updated over time. 

According to the theories of educational technology, evaluation ought to be a rather 
objective process, where learning results measured by some clearly defined performance 
standards are compared with the defined learning objectives (see e.g. Popham 1981). Our 
view is that this might be an ideal situation, at least in some cases. However, in practice, in 
NKI, and we believe in most other distance education institutions, the objectives defined 
during development of the course are seldom expressed in formal terms suitable for objective 
control in a strictly goal-related evaluation. Thus, the evaluation procedures tend to become 
more subjective than the ideal of the systems approach would prescribe. This seems to be a 
common observation. While criterion-referenced tests including performance standards 
originally were presented as a promising tool for securing accountability in education and thus 
constitute a simple and inexpensive tool for decision-makers, in practice over the years, the 
ideal theory has not lived up to expectations. For a discussion of performance standards and 
criteria, see Burton (1977) and Glass (1977). 

Concepts related to quality in business and industry 

As educational institutions in general have been influenced, often through governmental 
intervention, to adopt concepts and systems of quality developed in other sectors of society, 
and as these systems have been introduced in connection with the recent changes related to 
Norwegian distance education we shall look shortly at some of these concepts as well. 

Quality is most often defined 'fitness for purpose' related to the needs of the user/customer 
(Juran 1988), which indicates that quality depends upon a subject's view of what is the 
purpose of that phenomenon. In education the customer is not easily identified. The 
government pays, the immediate user is the student, secondary users are employers etc. 
Quality, thus, is a value judgement interpreted by different stakeholders, government, 
teachers, administrators, students, employers etc. On the other hand, to assure and assess 
quality we must have a clear notion of what it is. Another definition could be that the 'product 
comply with defined requirements'. Consequently, purpose and requirements, then, should be 
defined by the significant stakeholders. Bimbaum (1989) has stressed this diversity and 
pointed out three dimensions of quality in higher education: the meritocratic (the institution's 
conformity to professional and scholarly norms with the academic profession as reference 
group), the social (the degree to which the institutions satisfies the needs of important 
collective constituents) and the individualistic (the contribution the institution makes to the 
personal growth of students (from Van Vucht & Westerheijden 1993). 
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Some other important concepts are 'quality control', quality assurance', 'quality 
management' and 'quality assessment'. Quality control is defined in technical environments 
as: 'the operational techniques and activities that are used to fulfil the requirements for quality' 
(ISO 8402). Van Vucht & Westerheijden (ibid.) add that concerning higher education the term 
also includes the state control strategy concerning quality in higher education. Quality 
assurance is 'all those planned and systematic actions necessary to provide adequate 
confidence...' Quality management is defined as 'that aspect of the overall management 
function that determines and implements the quality policy'. The ISO document defines 
'quality audit', while Van Vucht and Westerheijden (ibid.) recommend the term 'quality 
assessment' as a better term in the field of higher education and describe it as 'a systematic 
examination to determine whether quality activities comply with planned arrangements and 
whether the product (the educational process) is implemented effectively and is suitable for 
achieving objectives' (Ibid. p. 12). 

Internal or external evaluation? 

There has been a lot of discussion in the educational community on who should carry out 
evaluation to guarantee maximum objectivity and usefulness of the results, see e.g. Scriven 
(1975). In our view, the question is more acute in situations where external bodies have the 
responsibility to accredit institutions and/or programmes according to certain criteria, and in 
situations where granting bodies wish to control the effects of money granted to certain 
programmes. According to Scriven (1975) there is no easy way around the question of 
objectivity. There is no way of guaranteeing that external evaluators are not biased, and there 
is a good chance that external evaluators are not able to go sufficiently into the institution and 
programme to come up with a useful evaluation design and get hold of the information of real 
importance for the evaluation. 

There are many reasons why evaluation and other measures to control and improve quality 
have been so important and have been taken seriously by distance education institutions: 

• Because of less immediate contacts between programme planners/teachers and students, 
there is a need of formalising evaluation of what is going on. 

• Distance education is normally organised in systems characterised by division of labour. 
This also makes formal evaluation more necessary. 

• Political purposes are often more explicitly stated in distance education, i.e. to reach 
particular target groups. Evaluation of results is therefore desirable. 

• Distance education is often innovative as regards pedagogical elements, media and 
technology, and organisation. Information on the effects is needed. 

• Distance education is often influenced by systems thinking and educational technology, 
where systematic evaluation plays an important role. 

• Distance education must demonstrate quality in order to meet the usual perception of being 
second-rate. 

Thus, a tradition of research and evaluation has emerged in both public and private distance 
teaching institutions, and this is often linked with systematic efforts towards improving 
quality and methods. The links between research and practical implementation are usually 
close, which also makes it easier to promote a 'practitioner-based' type of evaluation. For 
instance, the British Open University, had from the start a specific department responsible for 
research and evaluation. This example has been followed by many of the distance education 
universities established later in Europe and other parts of the world (see e.g. Schuemer 1991). 
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During the last years there has been an increasing emphasis that those directly involved and 
responsible should play an important role in the evaluation process. The idea is mainly that if 
the evaluation really is to have a positive impact on the improvement of the quality of the 
educational system, the practitioners must take part in the evaluation process (Thorpe 1988) 

As this article will illustrate, in NKI there are examples that evaluation may be undertaken 
by a member of the programme staff, by other member(s) of the responsible unit in the 
organisation, by another unit within the organisation or by external evaluators contracted by 
the teaching institution itself or by some external controlling or funding body. However, there 
is no doubt that the emphasis is on internal evaluation as the main measure for quality 
improvement. 



External evaluation of the organisation, programmes and courses. 

Norway was the first country in the world to regulate distance education by a specific 
legislation, the Act of 12 November 1948 concerning Correspondence Schools. According to 
the Act all correspondence schools and every single course must be accredited by the Ministry 
of Education before it can be put on the market. 

From 1949 until 1992 this evaluation was administered and carried out by a controlling 
body. The Correspondence Schools Council. The legislation was changed from 1993 when 
distance education became regulated as part of the Act on Adult Education. We will come 
back to this change later - as it has had considerable impact on external control and quality 
assurance in distance education in Norway. 

External evaluation on the institutional level 

By regular intervals the Council was supposed to conduct an intensive and inclusive 
evaluation of all distance teaching institutions. Thus, when subject to these evaluations, the 
total NKI distance teaching organisation and activities were examined. The reports were based 
on internal reports from NKI, meetings with key personnel and external analysis by the 
Correspondence School Council's representatives. The final reports from these evaluations 
dealt with marketing and economic systems, tutoring, support and counselling, qualifications 
for full and part-time employees, the quality of individual courses and study programmes that 
are offered, turn-around time of assignments, experimental activities and administrative 
routines, i.e. all the factors that were seen to affect the total quality of the educational 
programme offered. 

The institution gained quite some new and valuable insights through these intensive 
evaluation procedures. 

External evaluation of curricula and study programmes 

Traditionally, the Correspondence Course Council was not occupied with evaluation of the 
total study programmes or composition of courses. The evaluation concentrated on assessing 
the whole institution and the individual single courses. 

Norway also has a nation-wide system of recognition of college and university 
programmes and exams. In this connection NKI programmes on tertiary level have been 
externally evaluated. During this process, curricula, textbooks, exams, teachers and examiners 
are evaluated according to certain criteria defined by the Norwegian Universities. 
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The Correspondence Course Council was also responsible for evaluating each separate 
course. To carry out the evaluation of the courses, the Council appointed external subject 
experts. In our view, these procedures may have been justified at the time when they were 
initiated. However, in recent years the rigid system of external control was both by the 
Ministry and the institutions seen to have more weaknesses than strengths. Some of these are 
also discussed by Holmberg (1989). 



From external control to internal quality assurance 

Below we will try to account for the changes in legislation and external control regulations 
introduced to Norwegian distance teaching institutions in 1993. Although NKI always has 
seen its own internal evaluation procedures to be the backbone of quality work, there is no 
doubt that legislation with regard to external accreditation and control schemes do influence 
internal procedures. 

The Norwegian accreditation scheme 

There is hardly any doubt that the Norwegian Act concerning Correspondence Schools, the 
work of the Correspondence School Council and the Ministry's accreditation of 
correspondence courses and correspondence schools during the years has had a positive 
impact on the development of correspondence education in Norway. It would also be correct 
to argue that there are few examples of institutions that have operated with insufficient quality 
or doubtful business practices during the period in which the Act concerning Correspondence 
Schools has been in effect in Norway. 

The accreditation scheme ensured that each individual course was evaluated by 
independent consultants before it could be put on the market. As a result the schools 
established routines for quality assurance through the choice of authors, the editing work and 
in some cases the use of independent consultants before the courses were submitted to the 
Correspondence School Council for evaluation. Naturally the schools wanted to avoid the 
delays and negative rating associated with not having prepared their material sufficiently 
before the application for accreditation was submitted. The evaluation by independent 
consultants provided feedback that affected the actual development of the courses, so that 
changes and revisions could be carried out before the course was put on the market. 

Another factor that had a positive impact on quality assurance was the limiting of the 
process of accreditation to a period of three to five years. In this way Norwegian 
correspondence schools had to establish systematic routines for revision, reorganisation and 
updating of the course material and the development of new courses. 

In addition to the accreditation of individual courses, the Correspondence School Council 
was, as mentioned above, responsible for supervising the schools. 

It is likely that the authorities' follow up, supervision and economic support have played an 
important role in ensuring that distance education in Norway, which so far has mainly been 
represented by the independent distance education institutions, is recognised nationally and 
internationally for its high level of quality and competence. The situation is also a result of the 
institutions' willingness and ability to carry out systematic quality improvements, partly 
through research and development work. Some of this work at NKI will be described later in 
this article. 
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Why change a scheme that functioned well? 

The repeal of the Norwegian Act concerning Correspondence Schools, the placing of the 
approved, independent distance education institutions under the jurisdiction of the Norwegian 
Adult Education Act, and the delegation to the institutions of independent responsibility for 
their quality improvement work were each in many ways the result of a natural development. 

This change is partly a result of certain weaknesses in the old accreditation and supervision 
scheme and partly a result of the relatively high level of competence that the independent 
distance education institutions have attained. Furthermore, the technological and pedagogical 
developments in distance education and recent ideas about how quality improvement work 
should be organised and implemented in general have also tended to support the change. Last 
but not least, we should mention the close co-operation and trust between the institutions and 
the Ministry that have been established in recent years. All of these factors are closely related 
and the result is that evaluation and quality assurance have primarily become the institutions' 
own responsibility. 

Some weaknesses of the accreditation scheme under the Norwegian Act concerning 
Correspondence Schools can be related to factors involving the question of objectivity and 
how the institutions might utilise the findings of the Correspondence School Council's 
consultants. The question of objectivity cannot be resolved in any particularly easy way 
(Scriven 1975). It is not possible to guarantee that external appraisers will be able to delve 
sufficiently deeply into the course to give a reasonable appraisal or that they will be more 
objective than internal appraisers. Another problem is that those responsible for a course may 
loose their own initiative in the quality appraisal. For example, it is not very promising for an 
institution's quality improvement work if they regard an external approval as a guarantee that 
a product meets the quality requirements that they themselves ought to be ensuring. 

Due to questions of competence and objectivity, the Correspondence School Council's 
consultants had to be recruited from institutions that did not offer distance education. This 
meant that in many ways they were not very well qualified to evaluate correspondence courses 
and correspondence school education. Often they were poorly qualified to appraise what 
distance education entails on the whole and how the material in an individual course should 
function as one element in the total study programme. They could easily end up evaluating the 
actual course material for itself and comparing it directly with text books that were often 
developed for other target groups. 

There is little doubt that some consultants also started out with negative attitudes to 
distance education: "My field cannot be taught through a correspondence course." In some 
cases the consultant has had a more teacher-oriented attitude to education as opposed to the 
student and/or learning- oriented attitude, which has characterised distance education for a 
long time. These factors may have made it difficult to accept the teaching philosophy of the 
course. 

Whereas the consultants appointed from other institutions could be regarded as objective, it 
is clear that many, particularly in recent years, were not. The lack of objectivity might have 
been caused as other institutions regarded the distance education programme as a competitor. 
The recruiting base for certain types of education has been scanty at times, and some 
institutions may have been slow to adapt to demands from students or the authorities for less 
traditional and more flexible study programmes. 

It should certainly also be mentioned that external appraisals may cause those people 
responsible for the course to pay more attention to the consultant's appraisals than they do to 
the students' needs. In some cases, in fact, it turned out that the consultant demanded changes 
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in the course that the schools' course development personnel, on the basis of their knowledge 
of the field and of distance education, regarded as detrimental to the quality of the course. 

During the years that the Norwegian Act concerning Correspondence Schools was in 
effect, the schools underwent an extensive competence building. Since 1970 some of the 
schools, mainly lead by NKI, had been conducting significant research and development 
work, either individually or in collaboration, which resulted in internal competence building 
and external recognition. Under these circumstances, the external supervision of details 
seemed somewhat unnecessary. 

Taken together, these factors have brought the authorities to the conclusion that distance 
education should be regulated as a kind of adult education, and that accredited distance 
education institutions themselves, in collaboration with the authorities, are best able to take 
responsibility for quality control and quality assurance. 

Guidelines for quality standards in the Norwegian Association for Distance Education 
After the public regulation of distance education in independent institutions had been 
integrated into the Norwegian Adult Education Act, effective 1 January 1993, the 
responsibility for ensuring the quality of the learning material, the teaching and the practical 
implementation of the study programmes was delegated to the individually approved distance 
education institutions. The Norwegian Association for Distance Education (NADE) was 
requested by the Ministry to prepare guidelines for quality standards in distance education 
(see NADE 1996, Ljosa & Rekkedal 1993). 

In the documents related to the bill it was specified that the evaluation of quality ought to 
have a broad basis. This was been expressed as follows: 

• Quality assurance, follow up and control should be concerned with the total educational 
programme (learning material, teaching, guidance and follow up), and not with just part of 
the course material as is currently the case. 

• A course or an educational programme should be implemented according to a curriculum 
that is determined and described beforehand. 

• Marketing requirements have also to be taken into account. Since the schools' own 
advertisements and course descriptions are often the only information that the prospective 
students have when they register for a course, it is important that these advertisements are 
realistic and truthful. 

• The studies should normally be open to everyone, and the advertising ought to occur in a 
way that complies with this principle (Report to the Odelsting 1991-1992). 



NADE's Standing Committee on Quality and NADE’s Quality Standards 
NADE's Standing Committee on Quality was appointed late 1992. It is supposed to be the 
association's expert body in quality matters. It is supposed to work on matters involving 
quality criteria, quality standards, and quality assurance and improvement in distance 
education. The first main task of the committee was to develop Quality Standards for 
Norwegian distance teaching institutions. 
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The Quality Standards' function 

The individual distance education institutions differ greatly among themselves in purpose, 
type of activity, resources and size. It is therefore difficult to devise quality standards that are 
equally applicable in spite of these differences. NADE's standards are supposed to be 
recommendations and must give the individual institution sufficient freedom to define quality 
requirements on the basis of its own circumstances and possibilities. At the same time they 
must establish certain minimum requirements that are expected to be met if the institution is 
to be able to maintain a justifiable level of quality. 

The quality standards have both an internal and an external function. Internally at the 
individual institution, they are supposed to serve as guidelines for the institution's own quality 
improvement work. They should not relieve the institution of the responsibility for defining 
its own quality improvement policy and quality goals, nor should they prescribe in detail how 
the quality improvement work should be carried out. However, they should help the 
institutions to become more aware of their responsibility for quality in various areas, serve as 
an aid in the institutions' self-evaluation and define the agreed standards of NADE, which an 
accredited institution is expected to meet. 



Structure 

After consulting several reports and systems on evaluation and quality assurance, the 
committee took its point of departure in a matrix of problem areas for evaluation of education 
reported from the University of Lund in Sweden (Nilsson 1992). The NADE Quality 
Standards divide the distance education institutions' activities into four main categories: 

• Information and counselling 

• Course development 

• Course delivery 

• Organisation 

Each of these main categories is again divided into four «phases»: 

• Conditions and constraints 

• Implementation 

• Results 

• Follow up 

These are combined in a matrix of 16 elements called quality areas. For each of these 
quality areas certain factors have been specified, which should be examined when the 
institution evaluates its own quality. The quality standards that have been specified are 
grouped and numbered according to the areas and factors included in the matrix. 



The use of the Quality Standards in the individual institutions 

NADE's Quality Standards are recommended standards, and in most cases they are formulated 
rather generally so that they can be applied in very different institutions. 

The main reason for accounting for the change in control and accreditation schemes 
relating to distance education on the national level is that these governmental regulations have 
profound effects on the institutional level concerning formal quality assurance system and 
evaluation procedures. Experience so far has shown that the institutions have become subject 
to quite rigid external requirements for documentation, primarily in connection with reporting 
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to and supervision by the Ministry of Education, Research and Church Affairs. It is, of course, 
seen as an advantage for both the institution itself and the Ministry that quality improvement 
measures and quality assurance can be documented in a well-arranged and easily accessible 
form. However, it is a danger that the emphasis is turned from effective evaluation and quality 
work directly related towards improving the teaching system to formal systems suited mainly 
to satisfy external demands. NKI has a firm policy to devote the resources to the former. 



Internal evaluation 

As mentioned above, NKI has set up a separate unit for evaluation and research activities. 
While external evaluation has had some positive effects on the quality of the teaching system, 
external evaluation functions mainly as summative. 

Like most other distance teaching institutes NKI is mainly concerned with formative 
evaluation and assessment as a means for revision and development (see Scriven 1981). 

Internal evaluation is carried out by the unit for research and development both on its own 
initiative and on contract from the distance teaching unit. We agree completely with the 
principles put forward by Scriven (1975) for controlling evaluation bias that results should not 
be fed back only to the persons responsible for the programme or evaluation object, but also to 
their superiors. The normal procedures for internal evaluations carried out by the NKI 
research and development unit includes full internal publication. 

Evaluation is also initiated and carried out by the staff directly responsible for the 
administration of a certain course or programme. A programme administrator may develop 
and send questionnaires or carry out a telephone interview with students and tutors on a 
course etc. These evaluations tend to be more informal in design, development of instruments, 
and in reporting the results. From time to time the research unit is asked to give advice on 
such projects. 

At NKI we have experienced that formative evaluation of courses gives the best results 
when conducted by an internal specialist unit, not directly involved in the operation of the 
programme, but working in close co-operation with the programme staff. These evaluations 
should preferably be supplemented by evaluations done as a natural routine by the staff 
responsible for the programme, because everyone engaged in education (or any other activity) 
should see evaluation as an integral part of his/her work. 

It seems clear that evaluation activities initiated and carried out by the programme 
administration staff should be highly stimulated. They certainly give useful information. At 
the same time there is no doubt that the results, interpretations and conclusions often are 
biased. The results should be, and often are, discussed with people less involved and 
preferably trained in educational research and evaluation techniques. 

Sometimes there are good reasons for contracting external support in evaluation. It might 
be that it is of special importance to secure that the results are as unbiased as possible and 
accepted as unbiased by the external market. Sometimes we need extra manpower or 
additional competence. 

What or who is to be evaluated? In the introduction to this article we stated that we would 
concentrate on the evaluation of teaching systems, not student evaluation. However, as 
relevant student learning is the ultimate criterion for measuring teaching quality, student 
evaluation is, in fact, crucial also in systems evaluation. 
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Evaluating students 

At NKI student evaluation takes place continuously during the studies. Distance students 
submit their assignments and receive from their tutor comments, corrections and evaluations 
as subjective overall comments and a formal grade. Courses below tertiary level normally 
have a quite high submission density, every submission is expected to cover 10 to 25 hours of 
study. Tertiary level courses have much lower submission density, one submission is 
normally expected to cover 30 to 50 hours of study (Rekkedal & Blakstad 1994). Most 
courses end with a formal examination, either internal with internal examiners, internal with 
external examiners from public colleges/universities (according to requirements for national 
accreditation of college and university exams) or public nation-wide exams. 

Guidance to tutors concerning evaluation 

The following paragraphs represent nearly direct quotations of the advice given to NKI tutors 
to help them in evaluating and grading student assignments (Rekkedal 1997): 

When evaluating performance, the setting of grades can be a special problem. Written 
assignments in distance education must not be regarded as precisely parallel to a test in a 
conventional school. At school, a test is given mainly in order to examine what the students 
have learnt. In a distance learning course, the written assignments are an integral part of the 
tuition and the basis for the two-way communication which consists of the student's answers 
(with possible questions), and the tutor's corrections, comments and draft answers. We can 
therefore say that the teaching to which the student is entitled and which the distance course 
contains in total in each study unit is not completed until the student has received and gone 
through the tutor's comments and evaluation of the answers to the exercises. Consequently, it 
is a bit of a paradox when we assign grades for each answer in distance teaching - we give 
grades on the student's performance somewhere in the middle of the learning process of the 
task that is evaluated! This paradox is still more obvious in cases where the average of the 
grades given during the study constitutes the final grade for the course. 

Students, if they are aware of such procedures, may refrain from sending in answers when 
they meet problems in the course because they know this will lead to a weaker performance 
and subsequently bad grades. It is in just these situations that the student has a special need for 
the guidance and help the tutor can give. In the worst cases, such circumstances lead to the 
student's complete loss of courage, and to his giving up of the studies altogether. 

We have yet to find a satisfactory solution to this problem. However, we will indicate some 
ways in which it may be partly overcome. 

(1) The final grade on the course should not be worked out mechanically as the arithmetical 
average of the grades given on the individual study units. The final grade should instead be 
laid down following the judgement and evaluation of the student's total performance and 
development in the subject throughout the course. 

(2) To a certain degree, the grades on the written answers could be used as an individual 
pedagogical measure to motivate the student. Each student's answers on each study unit 
should be judged individually, as a slightly better or slightly worse grade than the 
objectively ‘correct’ one may have the best effect on the student's motivation and learning 
efforts. 

If possible, comments on each successive assignment should relate back to previous marks 
- this not only shows the student that the tutor is watching individual progress (and 
‘remembers’ work done, say, six months before) but may highlight study problems. A 
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steady falling-off of grades may indicate de-motivation - material too hard for the student? 
Or demanding new skills in which he or she is weak? 

(3) The student should be encouraged to ask questions if he or she is stuck or has, in vain, 
tried to solve an exercise - without being afraid that this will have negative consequences 
on the grade for the assignment or course. 

(4) When a student obviously shows defects in understanding the content of the learning 
material, one should ask him/her to review it. The revision may concern single exercises or 
whole study units. The tutor should give guiding comments and an evaluation of the work, 
but not give a grade. One has to give the student guidance on what should be revised, and 
to tell him/her that both the old and new answers to the written assignment should be sent 
in a second time. 

(5) If a student sends in answers far from complete (that is, neglects to solve essential 
exercises), the tutor should provide guiding comments, but not give a grade. Instead, the 
tutor should ask the student to complete the answers and also tell him/her that the entire set 
of answers must be sent in anew. A series of students encountering similar problems may 
highlight a structural or content weakness in the course material, and here NKI’s 
department for material development should be alerted. 

(6) The tutor should be cautious using the best grades; attaining the highest grade should be 
difficult to such an extent that the student would find this result extremely encouraging. 

To get the highest grade, the student must show ability in independent treatment and 
evaluation of the material. In addition, the answer must be well presented and lucid. If a 
student has solved all the exercises in an assignment, free of errors, and the tutor has put a 
‘correct’ mark beside each exercise without a comment, the student may be inclined to think 
that he/she has earned the highest grade. However, in many cases the tutor will give a slightly 
lower grade because he/she feels the right solutions can be even better done - with regard to 
content, logic, language, or even aesthetics. In order to avoid misunderstandings, one should 
comment along the way in the answers. In addition, the tutor should always state his/her 
reasons for the judgement. 

Concerning low grades and incomplete answers, as mentioned above, it should be noted 
that many students become discouraged by having to revise or repeat an assignment. It could 
also cause the student to drop out. Sometimes it is good advice to give a preliminary grade 
and let the student decide whether to revise or continue further in the course. 

In courses where one specific study unit is not a prerequisite for studying the next, the 
student could just continue after being informed about consequences for final grades or for 
earning a course certificate. 



Different types of assignments 

Most assignments for submission in NKI distance learning courses are of the essay type or 
different types of problem-solving assignments in subjects like mathematics, natural sciences 
and technology. Multiple choice and other types of objective tests have been introduced in 
some subjects - without much success. The main reason is that experience shows that making 
real good assignments with adequate ‘distractors’ demands a lot of time and competence. If in 
the near future we should use on-line scoring and feedback in courses offered 
«electronically», some submissions might consist of multiple choice type questions. 
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Examinations 

Assignments for submission are mainly measures for learning. Grades are meant to give 
information to help students to learn and to motivate them for further work. All NKI study 
programmes prepare for formal exams. Students who for some reason do not sit the exams, 
receive a transcript of study results when completing a study programme including final 
grades for each course. The transcript is a documentation of completion and might be taken 
into consideration by employers, but has no formal value. NKI study programmes may 
prepare for the following types of exams: 

1 . NKI internal exams. Many NKI study programmes have no parallel in the public school 
system. The tests are constructed by NKI staff (full-time or part-time) and are marked by 
examiners appointed by NKI. 

2. Public exams. The students study at NKI to register for public exaims. 

3. Exams at state colleges or universities - mainly for programmes where NKI Distance 
Education has a formal co-operation with the academic institution. 

According to Norwegian requirements for national accreditation of college and university 
exams, all exams at tertiary level must have two examiners, one internal and one external. The 
external examiner is normally teaching equivalent subjects at another institution. 

After having completed the distance study programme and having passed all the NKI 
internal exams related to the programme, the student receives a certificate from NKI. The 
certificate lists all courses completed including marks given by the distance tutor and exam 
results. 

To some extent examination results are used in course evaluation. In our experience, 
feedback from tutors and students in other forms than just grades given for individual 
assignments gives a better basis for assessing the teaching quality of a course or parts of a 
course. However, from time to time comparisons are made between examination results of 
distance students and full-time or part time students taking the same courses. Such 
comparisons may at least give some indication of the quality of the distance study material. 

In general, we are not in favour of putting too much emphasis on comparative studies. One 
has to take into consideration that students are self-selected with regard to the different forms 
of study, and that in general, part-time students drop out more frequently than full-time 
students. The raw results from some of our statistics indicate, however, that distance students 
achieve better results than part-time-on-campus students (Paulsen & Rekkedal 1990). 



Objects of systems evaluation 
The whole organisation 

As described above, the organisation is assessed by external bodies as part of the Ministry's 
general accreditation procedures of distance education. After the change in legislation in 1993, 
this external control has shifted to emphasise quality-assurance systems more than the 
assessing of quality as such - as a direct consequence of delegating responsibility to the 
institutions. Thus, it has become even more clear that it is the internal assessment activities 
which secure necessary development according to external and internal requirements. 
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Within the field of distance education NKI has carried out extensive survey research 
programmes to see whether the organisation functions according to its own goals and 
objectives. 

Recruitment and drop-out surveys have been considered to be of great importance. By 
means of recruitment surveys we obtain information to decide whether NKI recruits students 
according to national and institutional policy so that NKI should function nationally as a 
supplement to other institutions in securing possibilities for education despite of a person's 
geographical, social or educational background. 

We have carried out drop-out surveys to decide whether drop-out rates are acceptable in 
terms of internal and external criteria. The studies have been designed to find reasons for 
dropping out and relations between drop-out and personal, situational or course related 
factors. On the institutional level NKI has set specific goals to increase completion rates. E.g. 
on the institutional level we found in 1990 that the average level of course completions among 
students studying entirely at a distance was a little above 50%. A three-year plan was 
implemented to raise the average completion rates to a minimum of 60%. During 1993 to 
1 996 the average completion rate was at 69% (including students studying entirely at a 
distance and students in combined studies - distance study supplemented by local face-to-face 
classes). 

When the author chaired the Research Committee of the International Council for Distance 
Education some criticism was raised against the Committee's support of research projects on 
drop-out and completion rates. The basis for the criticism being that distance educators' 
preoccupation with drop-out problems lead to the traditional educational community seeing 
high drop-out rates as a special problem of distance education. At NKI we consider drop-out 
statistics to be one important indication of institutional success (although we agree that not 
every dropout necessarily proves there is a problem in the system). We do not believe that 
distance education generally has a more serious drop-out problem than other forms of part- 
time adult education (and as far as possible comparisons go this belief is statistically 
confirmed). As distance educators we ought to be proud of accounting also for our failures, 
and thus, be able to do something about them. 

In another dimension NKI carries out institutional evaluation every year in connection with 
recurrent long-time planning. During the planning activities an overall assessment of today's 
situation is undertaken and compared with a desired for situation three years ahead, and 
specific long- and short-time objectives for improvement are specified. 



Curricula and programmes, courses and media 

As mentioned, curricula and programmes are externally evaluated in connection with nation- 
wide accreditation schemes. However, if our aim really is to strive for higher quality, we have 
to take internal formative evaluation seriously. 

When launching new programmes, at least some evaluation activities are always initiated 
on the programme level. Later in this article we shall give an example of an intensive and 
continuous evaluation programme which took place in connection with the two first 
presentations of a new programme for managers in the field of health and social services. An 
external evaluator was contracted to follow up students and teachers through a whole year. 

The evaluator was given free hands in deciding evaluation design, procedures and 
instruments. The evaluator decided to base the approach in Stake's (1976) theoretical views on 
‘responsive evaluation 
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Printed material and two-way communication with a distance tutor is still the basis of the 
majority of NKI distance study courses. Thus, high quality of the printed material is a sine 
qua non for the total system to function satisfactory. During the development of the courses, 
evaluation and assessment take place in different ways. Some examples: 

An ‘instructional designer’ may be a member of the course development team and go 
through the material during the authoring phase acting as some kind of ‘surrogate student’ to 
assess the learning quality and suggest changes. 

Preliminary material is scrutinised by external subject consultants to check the academic 
quality. Until 1993, the material was controlled by the consultant from the State authorities 
before it was printed. The delegation of responsibility to the individual institution has 
increased need for adequate internal control of the quality of learning material. 

In some cases the course is printed in a preliminary version. With the help of postal 
questionnaires or more intensive methods, such as telephone or direct interviews, student 
attitudes and experiences are collected before final production. The distance tutors are asked 
to inform the administration about students’ difficulties, printing mistakes etc. as a matter of 
routine. 

During the last fifteen years other media, such as telephone, fax, computers (including 
computer mediated communication and lately the Internet and www), broadcast or taped 
audio and video have been introduced. When starting courses where new media and 
technology are involved, NKI often follows the programmes with specific research and 
evaluation projects. Some of these are shortly mentioned as case studies below. Similar 
procedures are normally followed to assess financial and educational aspects of new 
procedures for tutoring, student support and administration. 

The distance tutors represent the students’ main contacts during their studies. The quality 
of the tutors' work is extremely important for the total experienced quality of the system. For a 
long time the work of the tutors has been continuously surveyed. We have also established 
formal systems for evaluation of tutors. All new tutors have to go through a distance course 
(either by correspondence or via Internet and the www) covering important aspects of distance 
learning and tutoring. Prospective tutors are evaluated during this course to decide whether 
they possess the desirable qualities of a distance tutor (see e.g. Baath & Wangdahl 1976). 



Evaluation criteria 

Different criteria have been used for assessing quality. Some of these are mentioned below. 

Recruitment may be one criterion for deciding whether the institution, programme or 
course lives up to the intentions. It is a national policy in Norway that one should have equal 
opportunities for education and that education on all levels shall function as an instrument for 
democratisation and equal status in the society. It is a fact that adult education does not live up 
to this ideal goal. In many instances adult education actually increases differences by 
recruiting more students with a high prior level of education. With a formal institutional 
policy of supplementing public school by expanding opportunities for education, recruitment 
surveys are important. NKI has carried out a number of such surveys. 

On the other hand, studies of persistence are important in deciding which groups are best 
served. We also consider completion and drop-out rates to be important criteria for deciding 
the quality of study material and student services. 
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Grading, however, has not been found to function as the best evaluation criterion. The 
reason for this is that grading is normally subjective, during the continuous assessment often 
used individually to motivate and stimulate the students in their studies. Even as far as exams 
are concerned, we have to admit that evaluation is a result of the examiners' subjective 
decisions and not based on criteria-referenced measurements. And, as mentioned above, 
because of self-selection through recruitment and drop-out, comparisons between results in 
different student groups do not give valid data for assessing relative quality. 

To a large degree subjective information from students, tutors and experts is used for 
assessing the quality of evaluation objects: learning material, teaching processes and student 
services. 



Evaluation designs, methods and instruments 

At NKI we have to some extent differentiated informally between educational research, 
research-based evaluation and informal evaluation, projects carried out by personnel without 
specific competence in research methodology. 

Thus, both research and research-based evaluation and informal projects and continuous 
assessment projects have their place in the total system of evaluation. Over the years, we have 
carried out both experimental research, survey research, developmental testing and case 
studies in this connection. 

In line with the discussion above both quantitative and qualitative analyses are applied. 
According to our experiences it seems clear that in the case of evaluation of a certain product - 
whether a specific learning package or a complete programme, for revision purposes, 
intensive methods based on in-depth face-to-face interviews or interviews given by telephone 
are the methods which offer us more useful information than large scale studies based on 
postal questionnaires, which allow for more quantitative statistical treatment and analyses. 

Experimental research 

It seems clear that the results of most research projects may be used for evaluation purposes. 
During the years, we at NKI have carried out several experimental research projects on 
different aspects of distance education. 

The distance education scene is in many ways suited as a laboratory for experimental 
research on different methods and forms of teaching (and learning) with clearly defined 
variables, random selection of groups and measurable results, specifically concerning criteria 
related to study success, drop-out and completion of studies. 

A study based on an experimental design (including the testing of statistical significance) is 
a necessary condition for ‘demonstrating’ causal relationships between differences in 
treatment and results. If one wishes to examine whether one way of organising teaching is 
‘better’ than another, one may select two groups of students at random, carry out the two 
different teaching arrangements and control for possible sources of errors. If the experimental 
group achieves significantly better than the control group (and if the difference between the 
groups is larger than that one to be expected as the result of incidental variations), we may 
conclude that the experimental variable has had a positive effect. On the other hand, an 
administrator, needing information for rational decisions, might very well act on the basis of 
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data not statistically significant, as well as a variety of other more or less subjective 
information. 

In general, at NKI we have had very positive experiences that the results gained by means 
of experimental research projects have been as valid, in the sense that administrators and 
academics seem to emphasise that measures proven by experimental research to have positive 
effect are followed up and tend to maintained in the system. 

Experimental research may also have some disadvantages. There are many aspects of 
teaching and learning that do not easily lend themselves to experimental research. And when 
one has decided to carry out an experiment, it becomes important to complete the study under 
carefully controlled conditions, i.e. it becomes difficult to make changes in the teaching 
arrangements, even when it could be seen as a preferable solution. Sometimes, it could be 
more effective for the researcher to follow the process of teaching and learning and to respond 
with changes aiming at adapting the process to achieve optimal conditions for learning. Thus, 
the experiment may hinder rational and preferable changes for the benefit of the students 
taking part in the research study. Consequently, in many instances it is more convenient and 
more effective to carry out studies without the rigid controlling procedures of experimental 
research. 

All experimentally designed studies at NKI have been supplemented with interviews or 
questionnaires to examine other effects and results than those measured statistically as part of 
the experiment, such as achievements (measured in grades), study progression and drop-out. 
We have emphasised to collect extensive information for possible decisions concerning 
further developments and changes in teaching, organisation or administration. 



Some specific projects 
Experimental research 

As mentioned, NKI has put much emphasis on carrying out experimental research testing 
different hypotheses with regard to teaching techniques, as well as on the support and 
guidance of distance students. 

The experiments carried out at NKI have generally been carried out according to the 
following design (in some instances more than one experimental group have been included): 

R:E XI 01 
R : C X2 02 

R means that the students have been selected to the groups at random 
E is experimental group 
C is control group 

XI stands for the variable that is examined, while 

X2 stands for the treatment given to the control group 

01 and 02 stand for measurements of results, such as achievements/study 

success/completion. 

In the following we shall give some examples of experimental studies carried out at NKI. 
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Turn-around time 

Turn-around time is defined as the time elapsed between the student mailing the assignment 
until it is received by the student corrected and commented upon by the tutor. In the 
experiment, the turn-around time was reduced in an experimental group relative to a control 
group, without the tutor knowing who belonged to which group. According to information 
from the students on a questionnaire, the experienced turn-around time was reduced from a 
median of 8.3 days to 5.6 days. The result was a measured increase in completion rates from 
69% among the control students to 91% among the experimental students, the difference 
being statistical significant at .001 -level. According to student responses, the critical limit of 
turn-around time to be accepted as satisfying was one week, a conclusion which seems to be 
supported by later research (Rekkedal 1983). This experiment was chosen as basis for an 
international co-operative project organised by the ICDE Research Committee aiming to 
replicate previous research in different settings (Taylor et al. 1986). 



Systematic follow-up and introduction to study techniques 

In another experiment we tried to examine the effect of following up inactive students by a 
sequence of postcards and motivational letters. The sequence was started automatically when 
a student had not submitted assignments for one calendar month, and stopped when the 
student started to submit assignments again - or established contact with a student advisor. 
Differences in study activity between 240 students in each of one randomly selected control 
and one experimental group were measured. We found clear and statistical significant 
differences between the two groups after three months. During the third month of the 
experiment, 46% of the experimental and 3 1% of the control group students were active 
studying and/or in contact with their advisor (Rekkedal 1972a). 

The above project initiated another experiment involving 3 experimental groups and a 
control group to examine the effect of introducing an introductory course in study techniques 
and systematic follow-up of new students. This experiment concluded carefully that the 
course in study techniques combined with initial follow-up may reduce early drop-out 
(Rekkedal & Hallem 1975). The experimental studies also include a study on the use of pre- 
produced tutor comments and standard solutions in distance education, as well as a couple of 
media research studies, ‘the telephone as a medium for tutoring and guidance’ (Rekkedal 
1989) and ‘telefax for two-way communication’ (Rekkedal 1992). 



Personalising teaching in a large scale system 

This experiment arose out of experiences from the above mentioned projects, literature search 
and a specific aim of serving new students in a better way. 

During the planning stage we carried out some intensive group interviews with several new 
students. These interviews showed that the students seemed to be generally satisfied with their 
experiences in distance study. 

The students reported, however, one common difficulty: They were reluctant to contact the 
administration, the counsellors or their tutors when they met problems, and they were 
uncertain about whom to contact in order to seek advice on different problems. 

We felt that the rationalisation and ‘industrialisation ’ of distance education which seem to 
be necessary to cater for large student groups at acceptable costs in a ‘large scale ’ distance 
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teaching system probably result in a division of labour and a depersonalisation of the teaching 
processes. This situation seemed to cause greater difficulties for many students than we were 
aware of. 

In the early days of correspondence education, the schools were often established by 
enthusiastic persons stimulated by an idea or a specific situation. What these people lacked in 
formal pedagogical knowledge, they compensated by entrepreneurial vision and devotion to 
their work. As professional distance teaching organisations developed, some of what we 
earned in theoretical orientation and efficient systems might have been lost in reduced 
personal teaching/leaming relationships. 

Thus, we decided to design an experiment where we wished to personalise teaching by 
introducing what we called ‘the personal tutor/counsellor 

In the experiment one person took on the roles of different tutors in different subjects, as 
well as student advisors/counsellors. Within the responsibilities of this tutor we also tried to 
include other measures which we believed were important to help the distance learner to 
complete his/her studies successfully. In short, we tried to construct a system which would 
constitute a "new" way of organising the tutor's work during the initial phases of the study 
period, the main emphasis put on increasing the experienced quality of the didactic functions 
of the distance tutor and the two-way communication between the tutor and the student. In 
total, 1 0 different aspects of tutor-student interaction were included in the experimental 
variable. 

The students selected for the experiment were assigned to a personal tutor who followed 
them closely during the first 3 to 1 1 courses of the study programme for a certain 
qualification. By this formal change in organisation, a number of aspects of the tutor's work 
and the division of labour between the administration, the counsellors and the tutor were 
changed. 

We also selected a control group, which was not deprived of any of the services which had 
been introduced on the basis of theory and research during the preceding years, and which at 
that time constituted a normal part of the NKI distance teaching system. The main difference 
in the treatment of the experimental and the control group was that the experimental students 
were taken care of by one personal tutor combining administrative, teaching and counselling 
functions, which normally were divided between different persons, departments and 
specialists. 

Data were collected from the normal NKI files and study records, and minute records kept 
by the tutor on the communication with the students, follow-up letters, special measures, 
telephone calls etc. A small questionnaire was developed to assess the students' attitudes. It 
contained questions on correspondence study in general, the tutors' work and counselling to 
examine whether different aspects of being a correspondence student were experienced 
differently by the students in the two groups. 

The students were selected for the experiment for a period of 5 months. Data on study 
progress were collected at the time of completion of the first defined stage of the study 
programme, at the time of cancellation or exactly 8 months after enrolment (for students who 
neither had completed nor dropped out). 

It is easy to see that the experiment has some drawbacks concerning: 

• causal interpretation: it is uncertain (due to confounding of factors), what variable causes 

which effect; 
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• control of sources of error; and 

• generalisations to other systems. 

We were clearly aware of these problems when deciding to carry out the project. We 
simply wanted to examine a total system, instead of looking at isolated variables - the aim 
being to see if the new organisation "could produce a better total quality system". 
Consequently, we find that possible generalisations to other systems must origin in the basic 
ideas, rather than in specific findings. 

The experimental role of the tutor is described below. The experiment covered 1 0 different 
aspects of the tutor’s work. 

Figure 1. Aspects included in the ‘personal tutor/counsellor experiment’ 



Aspect 


Experimental group 


Control group 


1 . Tutor 


Same tutor during the first 3-1 1 
courses 


Different tutors in different courses 


2. Employment 


Permanently employed 
full office time 


Part-time employment at home, paid 
per assignment 


3. Tutoring/ counselling 


Same person responsible for all student 
communication 


Responsible for written assignments 
only, other persons for general 
counselling 


4. Turn-around time 


Assignments returned the same day 
from the school 


Assignments sent via the tutor's home 
address 


5. Study technique 


Same tutor teaches study techniques 


Specific part-time tutor in study 
techniques 


6. Follow-up of new students 


Tutor takes contact with all new 
students via mail or phone 


Automatic routines with form letters 


7. General follow-up 


Tutor takes contact with all inactive 
students via mail or phone 


Automatic sequence of form letters 


8. Telephone tutoring 


Students may phone the tutor. 
Tutor calls when needed 


No systematic use of telephone 
tutoring 


9. Tutor presentation 


Personal presentation with photo and 
phone numbers enclosed with the study 
material 


Presentation of each tutor enclosed 
with first assignment returned from the 
tutor in each separate course 


10. Pre-produced tutor 
comments 


Developed for all courses. 
Applied when needed 


May have been used by some tutors 



As a result of the experiment we hoped to find some answers to the following questions: 
Will this organisation have any effects on the number of students starting their studies 
(defined by submitting one or more assignments)? Will this organisation have any effects on 
drop-out rates during the early stages of study? Will this organisation have any effects on the 
students' pace of study and general study activity? Will this organisation affect the students' 
attitudes toward correspondence study in general and/or on specific aspects of correspondence 
study? What will the consequences of this new tutor role be regarding the organisation of 
other aspects of the distance education system as well as the operational costs of the system? 

We found marked and significant differences between the groups. 8 months after enrolment 
the experimental group had a significant higher rate of completion. The number of active 
students was comparable in the two groups, while the control group had a significantly higher 
number of inactive students. Similar results were found after one year of study. The 
experimental students were more active in their studies and had completed a larger number of 
study units and single courses during the experimental period. 
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Concerning ‘start/non-start’ we found no significant differences between the groups. The 
non-start rate was Tow’ in both groups, 9 and 13 percent respectively. Neither did we find any 
significant difference concerning lapse of time between enrolment and registration of the first 
assignment. 

In general, the experimental students reported more positive attitudes towards 
‘correspondence study in general’, ‘the study material’ and ‘the work on assignments’, aspects 
not included in the project. None of these differences were, however, statistically significant. 
This was hardly unexpected, as these aspects were not part of the experimental variables. 
Concerning ‘feelings of isolation’ we found lessrproblems in the experimental group, but 
again the differences were not statistically significant. This might be surprising as one could 
expect that the increase in personal communication we hoped to create by the integrated 
personal tutor/counsellor role would reduce possible feelings of isolation in the studies. 

On the other hand, we found clear and significant differences on the following areas: The 
experimental group students expressed more positive attitudes towards ‘the general quality of 
the tutor's work’, ‘assistance and support from the tutor’, ‘guidance and counselling from the 
institute, tutor or counsellor’ and ‘follow-up by the institute and/or the tutor’, and whether 
‘telephone tutoring is of any help in distance study’. 

The work on the ‘personal tutor/counsellor’ concept and this experiment resulted in NKI 
organising the initial phases of study according to these principles. We found that introducing 
the personal tutor/counsellor on a normal basis was both financially and pedagogically sound. 
However, during the last couple of years organisational changes, new media and some 
difficulties concerning cost and efficiency control of the tutor/counsellor role in NKI have 
gradually led to different organisational, administrative and teaching structures. Basic ideas 
from the project have survived and are implemented by emphasising the counselling functions 
of the distance tutor, whether part-time or full-time, and in connection with the introduction of 
new media and methods, such as systematic telephone tutoring, fax communication, computer 
mediated communication and two-way video conferencing. 



Survey research 

Concerning survey research, our first project carried out an intensive recruitment and drop-out 
analysis of one year intake to the distance study programmes (Rekkedal 1972b). This survey 
was followed up by two other surveys. The first looked in less depth into three years intake to 
see whether recruitment and completion trends had changed, the latter in relation to different 
procedures and student services introduced during the period. An other study compared 
recruitment to distance study and full-time engineering courses and followed the students 
through a four-year period and was ended with a completion/drop-out survey. This 
longitudinal study of distance students gave insights into differences between the students' 
plans, attitudes and expectations at enrolment and their experiences and attitudes given at the 
time of completion or cancellation (Rekkedal 1973b, 1976, 1978). 

The results from these studies may very briefly be summarised in the following statements: 

1 . Drop-out from distance study is larger in the initial phases of study than later. Thus, in 
preventing drop-out and helping student at risk, it is important to give specific attention to 
the first phase of studies. 
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2. A considerable number of students do not submit one single assignment. Thus, measures in 
the initial phase to support potential dropouts should be taken even before the first 
assignment is received from the student. 

3. There is a positive correlation between the age and success, i. e. older students do better 
than younger students. In general, according to our experiences, ‘adult’ students well 
established in a working situation with family and children succeed better than younger 
students with less vocational, family social and economic responsibilities. 

There is also a positive correlation between the level of previous education and all 
measures of success in distance study, and a negative relationship between time since last 
school experience and success. 

Thus, specific measures should be taken to support younger students, students with low 
previous education and students who have been away from school for some time. 

4. In accordance with the hypothesis of Bajtelsmit (1988; see also Raynor 1985) external 
factors related to vocational/professional and family responsibilities have a significant 
effect on the non-completion rate. NKI research suggests that when asking for secondary 
reasons for discontinuation, students also mention different aspects of teaching quality, 
institutional problems and learning difficulties, which supports the idea of an interaction 
between external and internal factors. 

5. Completion surveys carried out at NKI indicate a positive trend resulting in later cohorts of 
students showing higher persistence and lower drop-out rates, indicating that measures 
taken to improve teaching and support services do have measurable effects. 

Qvist-Eriksen (1979a, 1979b) carried out a large-scale evaluation survey of NKI combined 
courses (distance courses supported by local face-to-face teaching). Questionnaires were 
distributed to 142 teachers and 739 students. The teachers were asked to evaluate the quality 
of the materials, general aspects of the programmes, NKI services and procedures, roles of the 
distance tutor and local tutor etc. Similar aspects were covered by the students’ 
questionnaires. The results were used for revision of the system as such, specific programmes 
and procedures and course material. 



Technology and media research 

From the early eighties NKI has carried out research and tried out different kinds of 
technology. These projects have often been organised as ‘field studies’ of specific media and 
technologies, sometimes in co-operation with other Norwegian institutions, such as the 
Norwegian Telecom Research. This research has covered technologies such as telephone and 
fax applications (Rekkedal 1 989, 1 992), video, television and two way video conferencing 
(Holden 1992, 1993), audiographics (Rekkedal & Vigander 1990) and computer-mediated 
communication (Paulsen 1989a, 1989b, Paulsen & Rekkedal 1990a, 1990b). 



The EKKO Project 

The EKKO Project on computer mediated communication in distance education may stand as 
one example of testing of new media and communication technology. In spite of quite modest 
expectations of revolutionary changes when introducing new media in the distance learning 
system, we looked upon computer communication and conferencing as specifically promising. 
We found that many of the ideas of personal communication examined and supported in 
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previous experiments, such as ‘the personal tutor/counsellor’, could be built into a system of 
computer-mediated communication. Computer-mediated communication changes a lot of 
well-known variables in distance education (see for instance the discussion of computer 
conferencing as a ‘new (3rd. or 4th.) generation’ or ‘shift of paradigm’ in distance education; 
cf. Lauzon & Moore 1989, Garrison 1986, Holmberg 1990). Still, we believed that the 
medium could be introduced within the frames of a large scale system without too many 
difficulties. 

The NKI EKKO Project (the acronym EKKO stands for ‘electronic combined education ’ 
in Norwegian) started in 1986. The aim of the project was: to develop a computer-based 
conferencing system specifically designed for distance teaching and learning and to test it in 
different contexts to gain pedagogical and administrative/organisational experience within 
distance education based on computer conferencing - in order to install conferencing as a 
standard option for NKI distance students. 

The project has been organised as a long-term development task following these stages: 

1 . Introductory search in the field. 

2. Development of a specific conferencing system on the NKI mini computer, HP 3000. 

3. Pilot experiments with ‘on campus students’. 

4. Study visits to institutions in Europe and North America. 

5. Pilot test runs in distance learning. 

6. Introducing computer conferencing on a larger scale in distance learning. 

The EKKO Project has covered system development and theory studies as well as small 
surveys. The project has produced some research reports and many articles and conference 
papers. One specific student thesis analysed problems concerning adaptation and integration 
of distance education technology in the student's family (Bjorgen 1992). Morten Flate 
Paulsen, initiator of the EKKO Project, is working on a thesis on ‘pedagogical techniques for 
computer-mediated communication’ (Paulsen 1993). 

After nearly 10 years of research into computer conferencing, it is now offered as a 
standard option in several NKI courses and programmes, specifically at tertiary level. During 
these years we have continuously developed the computer and administrative systems and 
developed and tried out computer conferencing based distance learning in different subjects. 
Information has been collected systematically, both by the project manager and by the 
research unit. The methods applied have included continuous logging of communication 
activities, questionnaires and interviews with ‘electronic college’ students, prospective 
students, correspondence students (for comparisons) and teachers, and comparison of 
examination results. All data have been continuously published and reported. 

Through our research we have learned that computer-mediated communication as a 
technology is quite different from other media in distance education. However, we have not 
drawn the conclusion that computer conferencing in itself represents a new generation of 
distance education (Nipper 1989). Since 1994 we have changed from applying specific 
conferencing software to using the Internet and www. 

We have experienced that there are large differences in introducing computer conferencing 
in large-scale and small-scale distance education systems. Distance teaching by computer 
conferencing puts large demands on the tutor’s time and efforts. Increased costs in the 
teaching phase should ideally be compensated by reduced costs involved in course 
development. We have found that it is difficult to reduce development costs without reducing 
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the quality. Thus, computer conferencing in a large scale system may easily become very 
expensive. 

To be able to exploit the possibilities of group activities we have decided to organise the 
computer conferencing courses with fixed starting dates and fixed progression schedules, 
which has led to computer conferencing being less flexible them correspondence education. It 
seems that many students prefer the flexibility and individual freedom of correspondence 
study, while others stress the advantage of group communication in computer conferencing as 
a necessary condition for their participation in distance education. We see great challenges in 
trying to develop computer conferencing to take care of both the social needs of some students 
and the needs of other students who prefer the flexibility of individual study with possibilities 
of more efficient communication. 

Programme development and formative evaluation 

As mentioned previously in this article, NKI emphasises formative evaluation in connection 
with the development of new courses and programmes. The following is one example of a 
systematic evaluation project as part of specific programme development. Some years ago 
NKI started the planning for developing a two-year distance learning programme in 
« Management and Administration for the Health and Social Services Sector ». The 
programme in a preliminary version was ready for testing in 1990. During the development 
phase external experts had been contracted to evaluate the curriculum plans as they were 
finalized. Each single course was assessed and accredited by the government before it could 
be tested in the trial run (as part of the State-control routines at that time). The whole 
programme was submitted to the National Co-ordinating Body for University Studies, 
assessed and became formally recognised equal to one-year-full-time study in the national 
system. 

It was decided not to put the programme on the regular market during the first year. A 
contract was signed between one major hospital and NKI to carry out one full trial run. Later 
another trial group was accepted. During the test period major aspects of the programme, 
individual courses, study material, distance and face-to-face teaching/leaming processes were 
formally evaluated for updating and revision before the programme was offered on the regular 
market (Koch 1 990). 

An external evaluator was engaged to design and carry out the evaluation programme. The 
evaluator bases the design on Stake's (1976, 1981) responsive evaluation or transactional 
model for evaluation (Peersen 1992). 

The evaluation plan focused on three main areas: 

1. Frame/situational factors, e.g. study form and content, administration, student expectations 
and demands, teachers’ expectations of their contributions to the totality of the programme, 
the students' superiors' expectations, plans, time schedules, economy etc. 

2. The teaching/learning process, i.e. what is actually happening during the learning 
programme as effect of the ffame/situational factors. 

3. Results, e.g. the students' experiences of changed behaviour and attitudes in relevant areas, 
superiors' experience of changed behaviour, students' problem solving ability etc., as well 
as subjective experiences of course quality related to expectations and experiences. 

The results are caused by an interaction of frame-factor variables and process variables. 
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In line with Stake's theory, the evaluator wanted to find out what is of value to the students 
by gathering expressions of worth from the various participants. She stresses that the aim of 
the evaluation is to understand rather than to explain or to convey propositional knowledge. 

During the evaluation process she therefore applied different methods. Some examples: 

1 . Successive student questionnaires on expectations and experiences. 

2. Problem solving tests. 

3. Questionnaires to tutors. 

4. Continuous logging of experiences and happenings by tutors, on-the-job co-ordinator and 
programme co-ordinator. 

5. Interviews with the superiors of the students on their expectations and their perceptions of 
the students’ problem solving abilities in relevant areas. 

6. Observations of face-to-face sessions. 

7. Some selected students are logging continuously their experiences. 

8. Final in-depth interviews with students and superiors related to previous responses. 

The evaluation programme was meant to function as formative both for the trial period 
students and for revision of the total preliminary programme. 



Conclusion 

This article has tried to account for some NKI concepts of system evaluation, some practices 
and experiences. Some results from evaluation and research projects have also been presented. 
We have found it difficult to go into details of results because the different surveys, 
experimental research projects and evaluation projects cover many areas and dimensions. 
Describing specific results in more detail would go beyond the scope of this article. 

There is one important point we would like to stress. Systematic collection of data and 
reporting, in our experience, is the absolute fundamental basis for lasting changes and quality 
developments. We have seen that good ideas may be tried out and lead to a positive change in 
practice. However, if one does not take the time and resources to really examine and report the 
effect of specific changes and developments by methods which people trust, changes seem to 
have a tendency to be only short-lived. When people in the system are actively involved in the 
developments, and the developments are proven to be effective by accepted methods of 
research and evaluation, the developments tend to be followed up - and live. 
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1. Introduction 

Evaluation in the sense of "the systematic investigation of the worth or merit of some object" 
has several meanings within education. Its most common purpose is to grade students. Here 
we are concerned with "Programme Evaluation" (or the broadly synonymous term 
"Institutional Research"), where the focus is on how well a particular educational programme, 
curriculum or teaching method works, how it might be improved and how it compares with 
alternatives. 

When the Open University was created (it received its Royal Charter in 1969 and the first 
students began their courses in 1971) it was always intended that its activities should be 
closely monitored and evaluated. One can adduce several reasons for this: 

• The University had to be accountable to the Government which was the main source of 
funding. 

• The University was to be funded directly by the Department of Education and Science 
rather than through the Universities Grants Committee and therefore would be scrutinised 
as a unique institution rather than as just another university. 

• The University represented a major innovation in terms of teaching methods. 

• The teaching material would be available for public examination and would therefore need 
to be of a consistently high standard. 

• Most of the teaching and learning would be "invisible" because it would be carried out at a 
distance. 

• The students were to be admitted on a "first-come, first-served" basis, regardless of prior 
educational qualifications. It would be necessary to know just who the students were and 
what progress they made. 

Whatever the reasons, formative evaluation activities were already underway in 1970. Some 
of this took the form of developmental testing, with volunteers being recruited to try out the 
brand new foundation courses. Other researchers were studying those people who had 
enrolled on preparatory courses designed by the National Extension College, their purpose 
being to get an idea of who the first OU students were likely to be. The staff involved in this 
work were soon brought together in the new and rapidly expanding Institute of Educational 
Technology (IET). A large part of current evaluative research is carried out by members of 
IET which now has over fifty academic staff. However, not all of its members are involved in 
evaluation and much research that could be categorised as evaluation is in fact carried out by 
staff in other areas such as the Planning Division, the Business Development and Marketing 
Office (BDMO), the Faculties and the Regions. 
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In section two of this Chapter I have attempted to map out the types of evaluation that have or 
are being carried out anywhere in the University. In later sections I return to the questions of 
how and why the research agendas and styles have changed over time. 



2. Types of programme evaluation 

I. System evaluation 

a) Basic measures of activity 

Any system evaluation must begin with certain basic measures. How many courses 
have been produced? How many students are there? How many applicants had to be 
turned away? This data is drawn from administrative records and is presented 
regularly, often in the form of an annual report or, in the case of the OU, a 
"Statistical Digest". 

b) Measures of efficiency 

Allied to measures of activity come those of efficiency. How many students 
successfully complete the courses? What workload do they attempt? What is the 
throughput of students? Again this data comes from administrative records and is 
produced as part of regular monitoring procedures (McIntosh, Woodley and 
Morrison, 1980). 

Evaluation of this type has often moved beyond the descriptive to the examination 
of patterns and causes. For example, postal surveys have been carried out asking 
students why they dropped out of courses. However, these have tended to produce 
low response rates and answers of dubious validity. A more systematic approach 
involved a combination of methods including detailed statistical analysis, an 
understanding of the subjective process of "dropping-out" and an awareness of the 
different policy options and their likely impact (Woodley and Parlett, 1983). 

Other efficiency measures have centred on the question of cost-effectiveness. On 
the face of it distance education is a cheap teaching method, but just how cheap or 
whether it is cheap at all is still in dispute (Wagner, 1977; Mace, 1978; Rumble, 
1997). Disagreements among the economists centre upon difficulties in making 
comparisons with conventional institutions. How does one allow for the fact that 
distance students remain economically active while studying? Is it important that 
mature students have less years of economic activity in which to employ their new 
knowledge? 

c) Outcomes 

In the Open University the measurement of whether adequate learning has taken 
place has usually been left to the formal exams and assessment system. However, 
as noted later in this paper, there have been some attempts to measure the 
development of distance students as learners using study inventories and in-depth 
interviews over a period of several years. Also, in one isolated case, Open 
University economics students were compared with conventional students by 
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administering a standardised test of economic knowledge (Lumsden and Scott, 
1980). 

On some OU courses there are no formal exams and follow up surveys have to be 
carried out to see whether there have been appropriate changes in behaviour or 
attitudes. For example, in one study people who had bought a "Study Pack" on 
energy efficiency in the home were contacted several months later to see whether 
they had carried out the energy-saving measures that had been specifically 
recommended. In another study young mothers from deprived inner-city areas in 
Glasgow who had taken short "Community" courses were followed up to see what 
impact the courses had had on their lives (Fames, 1988). 

Mail surveys of Open University graduates have been carried out to measure the 
long-term outcomes for individuals following an extended period of OU study 
(Swift, 1982; Woodley, 1988). These studies have looked at personal, occupational 
and educational outcomes. It has allowed the University to say to what extent its 
qualifications have been accepted by other educational institutions for admission to 
post-graduate programmes and by professional bodies for membership. It has also 
recorded career changes and determined which, in the opinion of the graduates, 
occurred as a result of their studies. On a personal level, individuals have been 
asked about other changes such as growth in self-confidence and ability to 
communicate. 

In recent years more attention has been paid to the increasing proportion of students 
who gain a number of credits from the OU but leave before graduating. It is 
important to know whether these "dormant" students are leaving because the OU 
system is failing them or because they have already gained the benefits that they 
wanted. It is already known that many of these students have used their credits to 
transfer into full-time courses in conventional institutions of higher education. 

The recognition of OU qualifications has also been approached from the other 
direction. A survey of employers was carried out to establish the standing of the 
Open University degree and the acceptability of its graduates (McIntosh and Rigg, 
1979; Kirkwood et al, 1992). 

Distance teaching can have other outputs besides the more obvious ones and several 
of these have been looked at. These include the use, acknowledged or otherwise, of 
OU teaching materials on courses in other institutions (Moss, 1979; Glaister and 
Carr, 1986); the passing on of OU materials to other learners (Stainton Rogers, 
1984); and effects on the educational motivation of the children of distance learners 
(Fenster, 1982). 

d) Programme aims 

The Open University is committed to greater "open-ness" and to an increase in 
social equity. Consequently much of the system evaluation work has been devoted 
to investigating how far these goals have been achieved. Particular attention has 
been focussed on formerly disadvantaged groups such as women, ethnic minorities, 
working class people and those with low educational qualifications. As well as 
measuring their representation among the student body and looking at the progress 
they make on the courses, barriers to greater participation have also been examined. 
This has taken the form of surveys of the general public to determine the levels of 
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awareness and knowledge of the institution (Swift, 1980), surveys of people who 
sent for details but decide not to apply (Woodley and McIntosh, 1 977), and surveys 
of applicants who decline the offer of a place (Woodley, 1983). 



e) 





Policy Evaluation 

Formative evaluation in the policy area has sometimes taken the form of market 
research. Surveys of prospective students and employers have been carried out to 
measure the likely demand for possible new courses. Surveys of current students 
have also been used to sound out opinion on various policy options facing the 
University. For example, one study tested the reactions of Open University students 
to the possibility of reduced tutorial provision on higher level courses (Thorpe, et 
al, 1986). 

Policy Evaluation has also taken the form of monitoring. The Open University 
carries out regular surveys to monitor the financial impact of study on its students, 
thus gauging the effects of fee increases, changes in local authority assistance, the 
effects of its own financial assistance schemes, etc (Blacklock, 1 982). Other survey 
data on the ownership of televisions, cassette recorders, home computers, etc, can 
assist course planning (Grundin, 1983; Kirkwood, 1997). 

Research has also been used to evaluate the impact of policy changes. In one study 
researchers looked at the effects of an OU policy to "de-register" undergraduates 
who had made no progress with their studies over a number of years (Heron, et al, 
1986). The results showed that one unforeseen consequence of this policy was that 
the University had de-registered many of its own graduates. 

Finally, policy evaluation studies have also taken the form of experiments or pilot 
schemes. Perhaps the best known example involved the admission of school-leavers 
to the Open University. The University and the Government of the day disagreed 
about the suitability of the OU for school-leavers and so limited numbers were 
admitted on a trial basis and the outcome evaluated over several years before 
making a final decision (Woodley and McIntosh, 1980). In recent years there have 
been evaluations of "access" pilot schemes devised to prepare people for OU 
studies (Fung and Woodley, 1997) 



f) Organisational evaluation 

The Open University, just like any other large and complex organisation, can and 
has been evaluated in terms of its internal arrangements and procedures. In general 
terms this has involved scrutinising the financial management and general 
"organisation and methods" of the University. More specifically it has involved 
tasks such as the monitoring of tutors' marking patterns and the turn-around time 
for assignments. Evaluations have also been conducted into the course team 
approach to distance course writing (Nicodemus, 1992) 



II. Course Evaluation 



O 
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The second major strand of institutional research at the OU is "course evaluation", the aim of 
which is to improve the quality and effectiveness of the teaching and learning that takes place. 
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The evaluation of distance education teaching materials may seek to provide information that 
can be used during the process of developing or preparing materials or learning experiences - 
formative evaluation procedures, or information about how well the 'finished' instruction has 
worked in normal use - summative evaluation procedures (Scriven, 1967). In practice, it is 
often impossible to draw such a clear distinction, but it provides a useful way of considering 
methods of evaluation. 



Formative Evaluation 

a) Critical Commenting 

The great majority of OU teaching materials are prepared by course teams, and peer 
review of draft materials is common place. At an informal level this may simply 
involve one or more colleagues reading, listening to or looking at draft materials and 
providing comments in terms of the suitability of content and the style of 
presentation. On the other hand arrangements may be made for systematic critical 
commenting, with teachers or writers reviewing the materials prepared by all the 
others working on the same course or programme. Here there is the potential to 
improve not only individual teaching materials, but also the overall course of 
instruction. The reactions of colleagues can also be augmented by adopting the more 
formal procedure of inviting one or more experts in the field to act as Assessors to 
comment on the draft materials. 



b) Developmental Testing 

Developmental Testing takes place during the preparation phase and involves trying 
out draft teaching materials with students. The feedback obtained is used to guide and 
inform writers' revisions to the materials before they are committed to print or tape 
(Nathenson & Henderson, 1980). Such testing may range from a fairly informal 
student try-out of a single piece of teaching, to an elaborate procedure for testing draft 
materials for a whole course of instruction. 

Students study the draft materials in the usual manner and may be asked to undertake 
any other requirements, for example submitting assignments, attending tutorial 
sessions, etc. and possibly sitting an examination upon completion of the course. 

Their comments on and reactions to the teaching can be collected by means of 
questionnaires and/or interviews and observations and are fed into the process of 
revising the course materials for 'final' presentation. 

Experience of developmental testing at the Open University (Henderson, et al, 1983) 
indicates the strength of the procedure for the revision of materials within the overall 
structure of the course and that these can be of benefit to both course writers and 
students. It is, however, not particularly suitable for enabling major structural changes 
to be made to the course. In an attempt to allow for greater flexibility, a number of 
other procedures have been tried that are part formative and part summative 
(Henderson, et al, op.cit.). These involve collecting feedback from students and tutors 
on a short-term 'published' version of course materials, to inform revisions to be made 
for subsequent presentations. 
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Summative Evaluation 

The 'product' of course development in distance education is not just the materials that are 
delivered to students by one means or another. Rather it is the interaction of learners with 
those materials and other resources, possibly including tutors and fellow students (Thorpe, 
1979). Summative evaluation procedures are intended to provide information about a course 
or materials in use. 

a) Feedback from Tutors 

At the OU most of the part-time tutors are not involved in the development of 
teaching materials for the course(s) they tutor. Their role is to support courses by 
running tutorial sessions (face-to-face or by means of telecommunications), marking 
assignments, teaching at summer schools, etc. They clearly have more direct contact 
with students and mechanisms can be implemented to collect, on a systematic basis, 
evaluative comments from them on a range of issues. For example they can give their 
own reactions to the teaching materials, and also accounts of problems their students 
have encountered in their studies and assignments (Ryan, 1982). The experience of 
tutors in making the course work can provide particularly useful information for 
subsequent modifications to or adaptations of the teaching materials and instructional 
arrangements. 

b) Feedback from Students 

Feedback is gathered frequently from OU students while they are talking a course or 
shortly after its completion. In some cases it may be possible to implement some 
revisions during the presentation of a course as a result of students' comments, for 
example by providing a supplement to update information or to clarify a problem 
area. More frequently, the student feedback from one presentation of a course helps to 
determine revisions for subsequent presentations. After a course or programme of 
instruction has been presented in substantially the same form to many cohorts of 
students, feedback may be collected to inform decisions about remaking or replacing 
the course. Information gained by course writers about the success (or otherwise) of 
approaches and strategies employed in their distance teaching may prove to be of 
great value when they prepare further courses. 

Mail questionnaires are the most widely used method for collecting feedback from 
students and the types of information sought tend to fall into the following areas: 

• Extent of Utilisation. Students may be asked to indicate which parts of the course 
or programme they have studied, which components they have used, how much 
time they have spent on their studies, etc. They may also be asked to report on 
any problems they have encountered in obtaining the course materials or in 
gaining access to resources. 

• Overall View of the Teaching. Students may be asked to rate the teaching of a 
particular unit of instruction in terms of its interest, perceived relevance or 
usefulness, level of difficulty, etc. They may also be asked to rate individual 
components of a course (e.g. teaching text, audio-tape, etc) in terms of their 
relative usefulness. 
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• General Style of Presentation. Course writers may be keen to receive students' 
comments on the style of presentation, both in terms of layout, design, etc. and 
the coherence and clarity of the teaching. Perhaps more importantly, students 
could be asked to comment on the extent to which the teaching style or strategy 
had enabled them to become actively engaged in learning from the materials. 

• Specific Content Issues. It is important to know how well the teaching has 
achieved its aims and objectives. To this end, information about students' 
problems with key concepts, ideas and relationships, etc. can be of great value to 
course writers when it is time for revisions to be made. 



Cross-Sectional Studies 

Some evaluation work has involved study of a particular innovation or component used in a 
number of courses. The aim of such studies has been to draw out generalisations from the use 
of a particular aspect of the teaching, or to establish the effectiveness of a particular strategy 
or teaching medium. 

The role of particular course components has been of concern to the Open University and has 
been the focus of much research. Studies on the use of audio-visual media (Grundin, 1985), 
tutorials (Kelly, 1981; Kelly & Swift, 1983), and computer assisted learning (Scanlon, et al, 
1982) are examples of this type of work. The research has involved collecting information 
from students on their access to and use of particular components and resources, as well as 
eliciting their views on the contribution made to the teaching and the overall effectiveness of 
courses. 

The introduction of an innovatory teaching strategy has also given rise to evaluative research 
across a range of courses, for example the use of project work to encourage greater 
independence in distance education (Henry, 1979). 



Developmental Studies 

Some evaluation of teaching effectiveness has taken the form of research into the 
understandings developed by students as a result of their studies. Qualitative changes in the 
understanding of key concepts and relationships formed the focus of a study of Open 
University social science students (Taylor, et al, 1981a & 1981b), based upon a method for 
evaluating the content of students' learning developed in Sweden by Dahlgren (1978). 

The study was concerned with finding out not how much students know, but what they 
understand about particular concepts and principles. A group of university students taking an 
introductory social science course were individually asked a set of questions about key 
concepts and principles taught in the course. They were interviewed before commencing their 
studies and again after completion of the course. The findings of this study informed the 
writers of the replacement course, not only by identifying problem areas in the teaching, but 
also by illustrating the different levels of pre-course understandings that students were likely 
to have. 
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3. The evaluation agenda 

Most of the types of evaluation listed in the previous section have been carried out over the 
years by members of the Institute and the impression gained by the reader might be of a 
comprehensive, carefully integrated evaluation programme. However, the programme has 
been patchy and its development has been affected as much by external and internal pressures 
as by considered debate. 

In his review of access issues in the OU, West devised a typology which distinguished 
between the concepts of "Market", "Liberal", and "Radical" access policies (West, 1989). One 
can attempt to characterise different evaluation practices in a similar fashion. 

"Market evaluation" 

Here evaluation corresponds to the sort of research done by a profit-maximising 
manufacturing company. Students are seen as customers and research is devoted to 
maximising their number and their throughput. 

"Liberal evaluation" 

With this type of evaluation the researcher is the student's friend. Information is gathered on 
why people drop out, what they think is wrong with the courses, etc, so that the system can be 
made better for them. 

"Radical Evaluation " 

This can take several forms but essentially involves the evaluator in taking a critical stance 
concerning which people become students, what they are taught, how they are taught and the 
effects on their lives. 

Each of these evaluation strands exists within the OU and, somewhat confusingly, the same 
research study can be operating in all three modes. Nevertheless it is worth trying to document 
how the balance has shifted between the three over the years, and to consider the causes. 

"Market evaluation” was of key importance in the early years when the OU's viability was in 
question. Was it attracting sufficient students and was it producing graduates in a cost- 
effective fashion? This has been a continuing concern but takes on greater importance in years 
when the flow of new applicants slows down or when, as in 1990, the Department of 
Education and Science mounts a review to see how well the OU is performing. This is when 
evaluation clearly takes the form of self-defence rather than self-improvement (Woodley, 
1993). Here the role of the institutional researcher is to provide the information that the 
University can use to convince its paymasters that the high level of state-funding is justified. 
In the last year this has included attempting to demonstrate to the National Committee of 
Inquiry into Higher Education (the Dearing Report) that, among other things, the OU 
produces large numbers of graduates in Maths, Science and Technology, that its graduates 
perform well in the labour market, and that it attracts large numbers of students from 
educationally disadvantaged groups. 
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Market evaluation has grown in importance in terms of potential student demand. Increased 
attention is being paid to identifying possible courses that will attract large numbers of 
students, that will be sponsored by employers and that can command high fees. Members of 
IET have been involved in such work but it has increasingly come under the remit of the 
Business Development and Marketing Office. This office in turn has frequently used outside 
market research companies to carry out surveys whose aim is to find ways of maximising 
sales. 

.« 

"Liberal evaluation" has been almost synonymous with course feedback. When writing on 
this topic in 1990, 1 commented that it was surprising how little effort has been devoted to it 
in recent years, given that providing such feedback is always listed as one of IET's primary 
tasks (Woodley, 1991). In 1971 large numbers of students completed detailed questionnaires 
on each unit of each foundation course throughout the year. By 1 990 the main source of 
feedback came from the Annual Survey of New Courses. This was a single questionnaire 
completed by samples of students after the courses had finished and, as the title implies, only 
affects courses in their first year of presentation. I suggested that the lack of attention paid to 
this area had several causes. 

• The great increase in the number of courses being offered. 

• The decrease in staff numbers in IET. 

• The routine nature of this work means many researchers find it boring and there are 
no junior research staff to delegate it to. 

• The type of information collected does not often lead to simple messages for course 
improvement and the nature of OU course production means that very little actually 
can be changed in any case. 

• Academics who have been in the OU for a long time feel that they know how to 
produce good courses. 

However, the situation is now dramatically different. Every course is to be surveyed in its first 
year, in mid-term, and in the year before it is due to be re-made. This means that large 
samples of students will be surveyed on some 120 courses in each year - a massive increase in 
survey activity. The driving force has been the Teaching Quality Assessments (TQA's) carried 
out by the Higher Education Funding Council which look for evidence of student feedback 
systems and also systems to take action on this feedback. This has further enforced the 
"hegemony of survey methods" (Morgan, 1990), and has also effectively changed student 
feedback into a form of instrumental "market evaluation" 

In 1990 it appeared that developmental testing had virtually disappeared, mainly for the same 
reasons noted above, but also because of the cost and the difficulty of finding appropriate 
"students". However, it has been revived in recent years for courses using new teaching 
technologies, eg courses that involve "home computing", new subject areas, eg foreign 
languages, or new student groups, eg those taken sub-degree "access" courses. 

"Radical evaluation" has always existed on some level but enjoyed something of a resurgence 
in the early nineties. For example, following a Senate motion re-committing the University to 
the goals of widening access and of equal opportunities, much attention was paid to how 
many people from various disadvantaged groups such as ethnic minorities and the working 
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class have entered the OU, what courses they took and what progress they made as students 
(EO, 1990). Various pilot schemes were carried out and evaluated in the regions to see how 
things could be improved. However, the impetus seems to have been lost in this area with the 
closing of the Equal Opportunities Unit, its activities having been supposedly 
"mainstreamed". With the need to rapidly expand student numbers, especially in Science and 
Technology, the focus shifted away from characteristics to quantity. Now things may change 
again with the recommendation by Dearing that institutions that increase educational 
opportunities to disadvantaged groups should receive greater funding. If so "radical 
evaluation" may become "market evaluation". 

Some critical and theoretical research continues to exist and can be included under evaluation 
in its broadest sense. One strand focusses on the contextual nature of student learning. Some 
studies have concentrated on the interactions of learning and people's lives (Morgan and 
Holly, 1994), and others on how novice students acquire "academic" literacies (Lea, 1996). 
Critical evaluation of the curriculum has taken place in the context of the Equal Opportunities 
Report. Recommendations have been made concerning the content and style of all courses, the 
provision of course that focus on issues of discrimination and disadvantage, and appropriate 
staff recruitment and training. Elsewhere the critique has focussed on the ideological bias of 
the curriculum and the modes of delivery (Burt, 1989 & 1990). On a more general level, the 
critique has concerned distance education and society in general. One approach has been to 
consider modes of production and discusses the "Post-Fordist" shift from mass-production to 
flexible specialisation (Piore and Sable, 1984). 

In a recent paper, somewhat plaintively titled "Still seeking the silent revolution?", Morgan 
examines the extent to which qualitative methodology has been adopted by distance education 
(Morgan, 1997). He concludes that a silent revolution is "developing", but notes that the 
external environment and increasing managerialism often militate against it. I would contend 
that a silent revolution has occurred but that it exists alongside conventional practice with 
very little interchange between the two areas. The divide remains between "research" and 
"evaluation". What Fames termed "the new educational technology" (Fames, 1993) is 
increasingly evident in the research literature, it is taught within courses on distance 
education, and it is taken up by more and more PhD students, even within areas such as 
Information Technology. However, evaluation remains firmly grounded in empiricism and 
surveys. Interviews are seen as a means of constructing surveys, or a poor substitute when 
there is no time for a survey. For academics working in this area it can lead to a form of 
schizophrenia when one has to work in both paradigms. 



4. The Future 

The Open University, along with the rest of British higher education, is now expected to teach 
more students but with reduced funding. Furthermore, increased competition for students 
means that other institutions are making determined efforts to recruit mature students. These 
two forces are likely to lead to an even greater emphasis on market evaluation. If IET staff are 
not willing to do it then it will be done by other units such as BDMO or the Planning 
Division, or, as we have seen in the last year, it will be bought in from outside. 
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Given their status as tenured academics, researchers in IET will be able to pursue radical 
evaluation as long as they continue to receive the support of the OU's academic community. 
The fact that IET gained a top rating in two successive national Research Assessment 
Exercises has both helped to protect its position and to bring in research funds. However, 
institutional funding is likely to be cut and, as the first generation of staff begin to leave the 
OU, they are either not replaced or replaced by evaluators without the same security and 
freedom to determine the evaluation agenda. These pressures suggest that the cross-over in 
ideas from research to evaluation may be delayed still further. 
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